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Summary 


Background 

Navy  Personnel  Research  and  Development  Center  (NPRDC),  working  with  the  Naval  Sea 
Systems  Command  (NAVSEA)  and  Chief  of  Naval  Education  and  Training  (CNET),  completed  a 
program  of  research,  development,  and  evaluation  for  automating  classroom  activities  in  which 
interactive  electronic  technical  manuals  (lETMs)  are  used.  The  project  addresses  two  Navy 
training  problems:  (1)  the  need  to  utilize  emerging  technologies  to  improve  shipboard  maintenance 
performance  and  reduce  maintenance  costs,  and  (2)  the  need  to  improve  the  efficiency  of  the  Navy 
training  pipeline. 

Objective 

This  study  determines  the  costs  associated  with  the  lETM/AC  system  and  assesses  the  costs/ 
benefits  of  the  advanced  training  technology  implemented  in  the  GS  “C”  School  at  Service  School 
Command,  Great  Lakes,  IL. 

Approach 

The  cost/benefits  analysis  process  followed  guidelines  provided  by  Naval  Education  and 
Training  Program  Management  Support  Activity  (NETPMSA)  for  an  Abbreviated  System 
Decision  Paper  (ASDP).  The  ASDP  is  a  tool  that  is  currently  used  by  schools  in  the  CNET  chain 
of  command  to  obtain  support,  funding,  and  approval  for  small  information  system  projects.  The 
process  included  analysis  of  (1)  course  characteristics  and  potential  benefits  if  automated,  (2) 
trade-offs  in  course/school  management  requirements  for  course  conversion,  (3)  delivery  system 
capabilities,  (4)  interface  considerations  for  configuration,  (5)  non-recurring  and  recurring  cost 
estimates  for  implementation  of  the  lETM/AC,  and  (6)  quantifiable  and  non-quantifiable  benefits 
resulting  from  implementation  of  the  lETM/AC. 

Results 

Benefits  identified  during  this  analysis  include  improved  student  performance,  reduction  in 
student  training  costs,  and  reduction  in  course  reproduction  costs. 

Initial  data  indicated  a  slight  improvement  in  student  performance  scores  in  the  lETM-based 
courses  over  student  performance  scores  in  the  paper-based  courses.  These  preliminary  results 
suggest  that  introduction  of  the  lETM/AC  technology  and  subsequent  reduction  in  course  length 
has  not  had  an  adverse  effect  on  student  performance. 

The  elimination  of  7  training  weeks  from  the  GSE  pipeline  and  3  training  weeks  from  the  GSM 
pipeline  results  in  a  reduction  in  student  training  costs  of  over  $1,900,000  for  FY  95  and  FY  96. 

The  cost  savings  for  not  reproducing  the  paper-based  technical  manuals  used  in  the  GSE/GSM 
courses  is  estimated  at  over  $96,000  for  FY  96. 


Conclusions 


Adequate  planning  is  key  to  successful  implementation  of  the  lETM/AC.  Institutional  impacts, 
life  cycle  management,  roles  and  responsibilities  of  personnel,  user  training,  the  hardware  and 
software  acquisition  planning,  appropriateness  of  a  course  for  automation,  and  impact  on  the 
instructors  and  students  need  to  be  addressed  early  in  the  development  process. 

Benefits  in  both  preparing  and  delivering  instruction  may  be  realized  by  incorporating  the 
Training  Integration  Management  Software  (TIMS),  currently  under  evaluation  in  a  Diesel 
Division  course  at  Great  Lakes.  TIMS  will  accommodate  instructional  materials  from  a  variety  of 
sources  including  the  Authoring  Instructional  Materials  (AIM)  system.  Standard  Generalized 
Markup  Language  (SGML)  editors,  and  common  PC  word  processors. 

Recommendations 

The  GSE/GSM  demonstration  highlights  the  need  for  coordination  among  Navy  commands 
and  organizations.  As  the  lETM/AC  technology  evolves  and  costs  decrease,  more  systems  in  the 
fleet  will  be  targeted  for  lETM  development  and  more  formal  courses  in  the  schoolhouses  will  be 
identified  for  conversion  to  the  lETM/AC  format.  The  roles  and  responsibilities  of  cognizant 
organizations  should  be  well  defined  and  procedures  established  to  ensure  coordination. 

To  accommodate  the  Program  Objective  Memorandum  (POM)  budgeting  cycle,  the  current 
time  line  for  submission  of  both  an  Equipment  Facilities  Requirement  Plan  (EFRP)  and  an 
Abbreviated  System  Decision  Paper  (ASDP)  is  6  years  prior  to  project  implementation. 
Considering  the  rapid  advances  in  technology,  a  shorter  time  frame  between  project  initiation  and 
subsequent  funding  and  implementation  should  be  established. 

The  formal  Instructor  Training  Schools  should  be  revised  to  provide  training  in  topics  such  as 
computer  literacy  and  operator  skills,  techniques  for  designing  and  presenting  computer-based 
instructional  media,  use  of  automated  performance  assessment  tools,  and  technology-driven 
instructional  strategies. 

The  automated  classroom  technology  imposes  new  skill  requirements  for  management  and 
support  personnel.  The  possibility  of  establishing  a  formal  school  to  produce  a  Navy  version  of  an 
industry  certified  information  system  manager  and  network  administrator  should  be  investigated. 
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Section  One 


Introduction 


1.0  Introduction 


1.1  Problem 

The  Interactive  Electronic  Technical  Manual/ Automated  Classroom  (lETM/AC)  project  is 
a  joint  development  effort  between  the  Naval  Sea  Systems  Command  (NAVSEA)  and  the  Navy 
Personnel  Research  and  Development  Center  (NPRDC).  The  project  addresses  two  Navy  training 
problems:  (1)  the  need  to  utilize  emerging  technologies  to  improve  shipboard  maintenance 
performance  and  reduce  maintenance  costs,  and  (2)  the  need  to  streamline  the  Navy  training 
pipeline. 

In  addition,  the  project  addresses  the  following  Navy  training  development  and  delivery 
problems:  (1)  the  need  for  a  usable  system  to  develop,  revise,  and  deliver  multimedia  course 
materials;  (2)  the  need  to  accommodate  a  variety  of  media  (text,  graphics,  audio,  video,  computer- 
based)  in  the  classroom;  (3)  the  need  to  provide  instructors  with  a  paperless  delivery  environment 
that  eliminates  large  amounts  of  paper-based  training  materials  and  the  problems  associated  with 
updating  them;  and  (4)  the  need  to  enable  subject  matter  experts  and  instructors  with  minimal 
curriculum  development  or  training  delivery  background  to  develop,  modify,  and  deliver  integrated 
computerized  curriculum  materials. 

These  problems  have  been  addressed  by  combining  lETM  technology  with  an  integrated 
electronic  multimedia  editing  and  delivery  system.  The  automated  classroom  system  takes 
advantage  of  technologies  that  use  electronic  data,  permit  new  ways  of  creating  and  delivering 
classroom  instruction,  and  enable  standardized  ways  of  interchanging  various  electronic  forms  of 
texts  and  graphics.  The  lETM/AC  technology  is  in  use  in  the  Gas  Turbine  Technician  Electrical 
(GSE)  and  Gas  Turbine  Technician  Mechanical  (GSM)  courses  taught  in  the  Gas  Turbine 
Technician  Journeyman  Level  Accession  Training  (GS  “C”  School)  at  Service  School  Command, 
Great  Lakes,  IL  (SSC  GLARES). 

1.2  Objective 

As  information  technology  evolves  and  becomes  cheaper,  more  systems  in  the  fleet  are 
being  targeted  for  lETM  development  and  more  formal  courses  in  the  schoolhouses  are  being 
identified  for  conversion  to  an  lETM/AC  format.  Procedures  for  acquiring  and  logistically 
supporting  lETM/AC  technology  in  the  schoolhouses  are  essential.  Otherwise,  the  funding  and 
support  for  the  technology  may  not  be  there.  However,  a  methodology  for  analyzing  the  costs  and 
benefits  of  implementing  the  technology  needs  to  be  incorporated  into  the  decision  making 
process. 

This  study  determines  the  costs  associated  with  the  lETM/AC  system  and  assesses  the 
costs/benefits  of  the  advanced  training  technology  implemented  in  the  GS  “C”  School  at  SSC 
GLARES.  The  contractor.  Instructional  Science  and  Development,  Inc.  (ISD,  Inc.)  was  tasked  to 
develop  a  costs/benefits  model  to; 

1.  Identify  the  costs  associated  with  converting  conventional  instruction  to  automated 
instruction  using  the  lETM/AC  technology. 
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2.  Analyze  the  life  cycle  costs  for  implementing  the  lETM/AC  technology,  including  the 
hardware  and  software  components. 

3.  Measure  any  anticipated  or  actual  gains  in  efficiencies  achieved  through  the  imple¬ 
mentation  of  the  lETM/AC  technology. 

1.3  Background 

The  joint  lETM/AC  project  is  a  practical  and  expedient  means  of  combining  two  concurrent 
efforts,  one  by  NAVSEA  and  one  by  NPRDC,  to  accomplish  the  goals  of  both  projects.  In  August 
1994,  the  Implementation  Planning  Group,  co-chaired  by  the  Chief  of  Naval  Education  and 
Training  (CNET  N5123)  and  the  Naval  Sea  Systems  Command  (NAVSEA  04PT),  developed  a 
project  Evaluation  and  Implementation  Plan  (EIP)  for  the  joint  project. 

As  stated  in  the  EIP,  the  objective  of  the  project  is  to  develop  and  evaluate  an  automated 
classroom  that  incorporates  lETM  technology.  The  overall  goal  is  to  demonstrate  the  feasibility  of 
the  lETM/AC  technology  for  developing,  revising,  and  delivering  Navy  instruction.  NAVSEA  is 
responsible  for  the  technical  applications  of  lETMs  in  the  classroom,  while  NPRDC  is  responsible 
for  evaluating  the  efficiencies  of  an  automated  classroom,  examining  the  usefulness  of  specific 
training  enhancements  that  are  not  currently  available  in  the  lETM  automated  classroom,  and 
developing  a  curriculum  decision  algorithm.  Together  NAVSEA  and  NPRDC  will  develop  system 
specifications  and  recommendations  for  implementing  the  products  for  additional  courses  at  SSC 
GLARES,  other  Naval  Education  and  Training  Command  (NAVEDTRACOM)  schools,  and  the 
AEGIS  Training  Center. 

1.3.1  Scope  of  the  NAVSEA  Effort 

The  objective  of  the  NAVSEA  project  was  to  produce  an  lETM  database,  which  could  be 
used  aboard  ship  and  in  the  schoolhouse,  to  improve  shipboard  maintenance  performance,  decrease 
maintenance  costs,  and  reduce  the  volumes  of  paper  technical  manuals.  The  lETM  database  was 
designed  to  support  a  dynamic  interactive  technical  training  environment  to  enhance  training  and 
reduce  course  length. 

As  stated  by  Alligood  (1994),  NAVSEA  selected  the  gas  turbine  systems  for  lETM 
database  development  for  the  following  reasons: 

1.  “Gas  Turbine  Propulsion  Ships  will  comprise  one-third  of  the  Navy  at  this  decade’s 
end-wide  applicability!”  (pg.  1). 

2.  There  is  a  need  to  improve  maintenance  performance  and  reduce  maintenance  costs. 
Gas  turbine  systems  are  the  most  expensive  set  of  systems  in  the  Navy  to  maintain  in  terms  of 
ownership  costs  and  ship  man  hours.  lETM  demonstration  projects  have  shown  more  accurate 
troubleshooting,  better  diagnostics  by  technicians,  less  time  to  perform  maintenance  tasks,  and 
the  ability  of  inexperienced  technicians  to  perform  maintenance  at  a  level  comparable  to  experi¬ 
enced  technicians  when  using  an  lETM. 
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3.  Since  all  of  the  gas  turbine  systems  are  mature,  the  lETM  would  be  based  on  legacy 
data.  The  development  approach  would  include  all  technical  information/documentation  for  the 
systems  as  well  as  the  GS  “C”  School  training  curricula.  Gas  turbine  systems  encompass  multiple 
equipments,  with  numerous  configurations,  installed  in  multiple  ship  classes. 

4.  There  is  a  need  to  reduce  the  GSE  and  GSM  training  pipelines  to  avoid  and/or  save 
training  costs.  The  GSE  training  pipeline  (including  console  operator  training)  is  approximately 
23  weeks,  while  the  GSM  training  pipeline  (including  console  operator  training)  is  20  weeks.  The 
lETM  demonstration  for  the  AN/BSY-2  Combat  Systems  is  an  example  of  the  potential  training 
benefits  (33%  reduction  in  the  number  of  training  courses  and  a  28%  reduction  in  the  number  of 
training  days)  that  can  be  realized. 

Under  the  direction  of  NAVSEAs  Technical  Data  Division  (SEA  04TD),  the  following 
organizations  participated  in  the  development  of  the  GS  lETM  database:  (1)  the  prime  contractor, 
Lockheed/Martin  (formerly  Martin  Marietta,  Ocean  Systems  Division),  (2)  the  Gas  Turbine 
Systems  In-Service  Engineering  Activity  (GS  ISEA),  Naval  Surface  Warfare  Center,  Carderock 
Division,  Philadelphia  Detachment  (NAVSSES),  and  (3)  New  Technologies  Division  personnel 
and  GS  “C”  school  instructional  staff  at  SSC  GLARES.  The  GS  lETM  development  process 
followed  the  approach  Lockheed/Martin  used  in  developing  the  AN/BSY-2  Combat  Systems 
lETM.  The  process,  as  illustrated  in  Figure  2,  takes  advantage  of  best  commercial  practices  for 
data  conversion  and  CALS  (Continuous  Acquisition  and  Life  Cycle  Support)  compliant  tagging  in 
SGML  (Standard  Generalized  Mark-up  Language)  and  best  commercial  practices  for  data 
translation,  integration,  hyperlinking,  and  display. 

Over  1 10,000  pages  of  technical  information/documentation  were  converted  and  SGML 
tagged  during  the  development  of  the  GS  lETM.  NAVSEA  developed  the  Document  Type 
Definitions  (DTD).  The  DTD  consists  of  a  set  of  SGML  tags  (developed  in  compliance  with  MIL- 
M-28001B,  August  1993)  that  were  used  to  tag  data  to  the  paragraph  level  as  the  smallest  object 
of  data  converted.  Diagrams,  figures,  and  tables  were  also  SGML  object  tagged.  The  converted 
data  was  uploaded  into  an  object  oriented  common  lETM  database  using  relational  database 
management  software.  The  common  database  consists  of  two  files-the  file  containing  all  of  the 
text  and  graphics  and  the  object  ID  file  containing  the  SGML  tags. 

To  create  lETM  functionality  for  performing  shipboard  maintenance,  view  packages  for 
general  knowledge,  troubleshooting,  maintenance,  and  operations  were  developed  by  subject 
matter  experts  (SMEs).  Each  view  package  is  a  script  that  sets  the  flow  from  one  object,  such  as  a 
paragraph,  graphic  or  table,  to  another  that  the  SMEs  consider  the  best  approach  to  accomplish  a 
procedure  or  task,  such  as  troubleshooting  a  specific  piece  of  equipment.  The  SMEs,  consisting  of 
personnel  from  SSC  GLARES  (New  Technologies  Division  and  GS  “C”  school  instructors)  and 
NAVSSES  (GS  ISEA),  were  trained  in  view  package  authoring  techniques  by  Lockheed/Martin 
staff.  Development  of  the  view  packages  was  not  difficult,  but  it  was  labor  intensive  and  required 
several  months  of  effort. 


5 


Figure  1.  The  gas  turbine  lETM  development  process. 


The  view  packages  were  authored  by  linking,  in  script  format,  the  object  file  IDs  (SGML 
tags)  and  applying  content  data  model  (CDM)  SGML  tags  (defined  in  MIL-M-87269,  November, 
1992).  View  package  scripts  contained  commands  to  link  and/or  launch  to  other  applications  such 
as  video,  audio,  animations,  and  other  documentation  on  compact  disk.  The  scripted  information 
was  uploaded  via  an  ASCII  text  editor  to  the  common  lETM  database  to  provide  positive  linkage 
of  data  in  native  SGML  format.  Lockheed/Martin  then  processed  the  database  through  a  translator 
for  presentation  using  commercial-off-the-shelf  (COTS)  hypertext/hyperlink  software  (InfoAccess 
GUIDE).  The  lETM  database  was  mastered  on  CD-ROM  for  distribution  to  the  fleet  and  the 
schoolhouse  to  be  displayed  on  COTS  personal  computer  hardware. 

A  Memorandum  of  Agreement  between  NAVSEA  and  SSC  GLARES  gave  NAVSEA  the 
responsibility  for  developing  the  electronically  enhanced  Instructor  Guides  (IGs)  and  Trainee 
Guides  (TGs)  for  inclusion  in  the  lETM  database.  In  addition  to  the  view  package  development 
effort,  personnel  from  the  New  Technologies  division  and  the  GSE/GSM  instructional  staff  were 
also  rewriting  the  conventional  paper-based  curricula  for  inclusion  in  the  lETM  database.  Rather 
than  revising  the  courses  in  Word  Perfect  and  then  converting  them  over  to  AIM  (Authoring 
Instructional  Materials),  the  Navy’s  current  system  for  authoring  PPP-based  (Personnel 
Performance  Profiles)  instruction,  the  revised  IGs  and  TGs  were  authored  using  AIM.  The  AIM 
databases  were  provided  to  Lockheed/Martin  to  extract  and  convert  the  training  materials  to  SGML 
format  for  inclusion  in  the  lETM  database.  The  IGs  and  TGs  have  the  capability  to  search  and 
launch  to  associated  multimedia  materials  such  as  video,  audio,  and  graphics  for  classroom 
presentation  that  are  not  in  the  lETM  database. 
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NAVSEA  acquired  and  installed  the  hardware  and  software  for  five  automated  classrooms 
in  the  GS  “C”  School  to  evaluate  the  effectiveness  of  the  lETM  technology  in  the  formal  school 
setting. 

1.3.2  Scope  of  the  NPRDC  Effort 

The  objective  of  the  NPRDC  effort  was  to  develop  a  prototype  system  for  developing, 
revising,  managing,  and  delivering  multimedia  course  materials  that  is  tied  to  the  CNET 
curriculum  development  and  maintenance  infrastructure.  NPRDC  developed  a  software  system 
architecture  to:  (1)  allow  instructors  and  other  schoolhouse  staff,  with  minimal  training,  to  develop, 
manage,  organize,  and  deliver  different  forms  of  electronic  training  materials;  (2)  accommodate  a 
variety  of  media  (computer-based  text,  graphics,  audio,  video);  (3)  provide  instructors  with  a 
paperless  delivery  environment,  and  (4)  ensure  compatibility  with,  but  not  dependence  upon,  the 
AIM  system. 

To  accomplish  these  goals  and  minimize  format  differences  between  AIM  and  COTS 
software,  the  NP^C  software  architecture,  referred  to  as  the  Training  Integration  Management 
Software  (TIMS),  provides  several  features  to  augment  the  NAVSEA  lETM  classrooms.  Currently, 
CNET  schools  prepare  task-based  instructional  materials  using  Word  Perfect  word  processing 
software  and  PPP-based  instructional  materials  using  AIM.  Since  the  NAVSEA  lETM  process 
requires  SGML  format,  the  IGs  and  TGs  in  the  AIM  database  or  in  Word  Perfect  format  need  to  be 
SGML  tagged  for  inclusion  in  the  lETM  database.  TIMS  provides  a  format  neutral  processing 
capability  to  electronically  present  instructional  materials  from  different  source  formats  (i.e.,  AIM 
or  Word  Perfect)  in  the  automated  classrooms.  It  is  a  software  program  that  uses  Windows 
interprocess  communication  mechanisms  to  link  media  objects  (e.g.,  graphic,  text,  lETM 
references,  animations,  audio,  video)  without  regard  to  their  data  format.  It  is  designed  to  allow 
curriculum  developers  and  instructors  to  link  media  objects  to  IG  paragraphs  without  SGML 
tagging. 

TIMS  has  three  modes.  Each  mode  supports  different  functions  that  Navy  personnel  must 
perform.  In  the  annotation  mode,  it  allows  curriculum  developers  to  attach  current,  standardized 
annotations  (such  as  lETM  references)  that  all  of  the  instructors  will  present  in  the  classroom.  In 
the  personalization  mode,  it  allows  an  individual  instructor  to  attach  personal  annotations  (text 
notes  as  well  as  other  media)  to  IG  paragraphs  in  the  Related  Instructional  Activity  (RIA)  column 
for  use  in  the  classroom.  In  the  classroom  mode,  it  supports  the  automated  classroom  presentation 
by  including  a  programmable  remote  control  capability  to  allow  the  instructor  to  control  classroom 
presentation  from  anywhere  in  the  room.  It  also  allows  the  instructor  to  digitally  broadcast  video 
displays  to  student  stations  without  the  analog  switches  used  in  the  NAVSEA  lETM  classrooms. 

In  addition  to  the  demonstration  of  TIMS,  NPRDC  was  also  tasked  to  evaluate  the 
effectiveness  and  efficiencies  of  the  lETM  automated  classrooms  at  SSC  GLARES.  The  evaluation 
variables  include:  (1)  a  comparison  of  student  performance  scores  between  the  paper-based  and  the 
BETM-based  GSE/GSM  courses,  (2)  a  comparison  of  course  content  and  length  between  the  paper- 
based  and  the  lETM  based  GSE/GSM  courses,  (3)  instructor  and  student  control  issues  in  the 
lETM/AC  classroom,  (4)  the  impact  of  the  lETM/AC  technology  on  management  and  the 
institution,  and  (5)  the  impact  of  the  lETM/AC  technology  on  costs  and  time  associated  with 
running  a  traditional  classroom. 
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The  costs/benefits  analysis  presented  in  this  report  is  one  aspect  of  the  evaluation  that 
NPRDC  is  conducting  of  the  lETM/AC  implementation  at  the  GS  “C”  School  at  SSC  GLARES. 
NPRDC  may  use  the  results  of  this  study  to  assist  in  developing  the  curriculum  decision  algorithm 
and  providing  implementation  recommendations. 

1.4  Report  Organization 

The  remainder  of  this  report  is  organized  into  the  following  sections. 

1.  Approach— a  discussion  of  the  review  of  literature  and  a  summary  of  the  rationale  and 
methodology  developed  in  the  study  for  analyzing  the  costs  and  benefits  of  implementing  the 
lETM/AC  technology. 

2.  Costs/Benefits  Analysis  Process-a  detailed  description  of  the  costs/benefits  analysis 
process  we  develop)ed  for  decision  makers  considering  implementation  of  the  lETM/AC  technol¬ 
ogy. 

3.  Costs/Benefits  Model-the  results  of  the  costs/benefits  analysis  for  implementing  the 
lETM/AC  technology  in  the  GS  “C”  School  at  SSC  GLARES. 

4.  Discussion — a  discussion  of  the  R&D  effort  and  the  unplications  for  future  implemen¬ 
tation  of  the  lETM/AC  technology. 

5.  Recommendations-a  list  of  suggestions  for  implementing  the  lETM/AC  technology 
in  NAVEDTRACOM  schools  in  the  future. 
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Section  Two 


Approach 


2.0  Approach 


2.1  Review  of  Literature 

A  review  of  cost  analysis  techniques  was  conducted  to  identify  the  best  approach  for 
analyzing  the  costs  and  benefits  associated  with  the  lETM/AC  technology.  Within  the  socio¬ 
political  context  of  educational  evaluation,  the  term  cost  analysis  refers  to  the  use  of  a  broad  set  of 
techniques  for  evaluation  and  decision  making,  including  cost-effectiveness,  cost-benefit,  cost- 
utility,  and  cost-feasibility  (Levin,  1983).  Although  these  terms  are  used  interchangeably,  each 
approach  has  specific  applications.  The  following  definitions  are  offered  by  Levin  (1983): 

1.  Cost-effectiveness  analysis  refers  to  the  evaluation  of  alternatives  according  to  both 
their  costs  and  their  effects  with  regard  to  producing  some  outcome  or  set  of  outcomes.  To  be 
selected,  an  alternative  must  provide  either  maximum  effectiveness  for  a  set  level  of  cost  or  least 
cost  for  a  set  level  of  effectiveness. 

2.  Cost-benefit  analysis  refers  to  the  evaluation  of  alternatives  according  to  a  comparison 
of  both  their  costs  and  benefits  when  each  is  measured  in  monetary  terms.  To  be  selected  an  alter¬ 
native  must  show  benefits  in  excess  of  costs. 

3.  Cost-utility  analysis  refers  to  the  evaluation  of  alternatives  according  to  a  comparison 
of  their  costs  and  the  estimated  utility  or  value  of  their  outcomes.  This  technique  permits  the  use 
of  a  wide  range  of  qualitative  and  quantitative  data  to  be  included  in  the  evaluation. 

4.  Cost-feasibility  analysis  refers  to  the  method  of  estimating  only  the  costs  of  an  alterna¬ 
tive  to  determine  whether  or  not  it  exceeds  the  budget  and  other  resources  that  are  available. 

The  objectives  of  this  study  fit  within  the  definition  of  cost-benefit  analysis.  A  definitive 
approach  to  cost-benefit  analysis  is  provided  in  the  DoD  Handbook  for  Economic  Analysis  (2nd 
edition).  This  handbook  provides  detailed  guidelines  on  the  process  and  procedures  for  estimating 
costs,  judging  benefits,  and  comparing  alternatives.  However,  the  data  requirements  for  performing 
an  economic  analysis  are  far  more  stringent  than  the  data  that  was  available  from  this  project.  For 
example,  sufficient  data  to  specify  alternative  configurations  and  derive  the  costs  and  benefits  for 
each  of  the  alternatives  was  not  available  at  this  point  in  the  ongoing  evaluation.  In  addition,  the 
effort  required  to  perform  an  economic  analysis  was  beyond  the  scope  of  the  effort  proposed  for 
this  study.  Since  following  the  traditional  cost-benefit  analysis  techniques  was  not  feasible,  we 
looked  for  another  framework  for  analyzing  the  costs  and  benefits  of  implementing  the  lETM/AC 
technology  in  NAVEDTRACOM  schools. 

2.2  Rationale 

After  discussions  with  Navy  personnel  in  different  organizations,  we  selected  the 
requirements  of  the  Abbreviated  System  Decision  Paper  (ASDP)  as  the  basis  of  our  approach  for 
analyzing  the  costs  and  benefits  of  implementing  the  lETM/AC  technology.  The  ASDP  is  a  tool 
that  is  currently  used  by  schools  in  the  CNET  chain  of  command  to  obtain  support,  funding,  and 
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approval  for  small  information  system  projects.  Since  the  format  for  an  ASDP  includes  a  statement 
of  costs  and  benefits,  it  is  relevant  to  the  issues  in  this  study. 

In  February  1994,  the  Naval  Education  and  Training  Program  Management  Support 
Activity  (NETPMSA,  Code  0634)  issued  a  revised  set  of  guidelines  for  automated  data  processing 
(ADP)  personnel  and  non-ADP  end  users  to  use  in  preparing  an  ASDP.  These  guidelines  amplify 
the  information  contained  in  SECNAV  Instruction  5231. 1C,  Appendix  A  to  Enclosure  (2),  and 
CNET  Instruction  5231.  IB.  It  specifies  the  abbreviated  life  cycle  management  (LCM) 
documentation  needed  for  acquisition  of  microcomputers,  local  area  networks,  and  multimedia 
hardware  and  software.  It  provides  guidance  from  project  initiation  through  funding  and  project 
approval. 

In  accordance  with  the  guidelines,  an  ASDP  consists  of  the  eight  major  sections 
summarized  below; 

1.  Need.  Outline  the  need  for  automation.  Define  the  scope  of  the  project.  Summarize  the 
functional  requirements,  including  local  area  network  (LAN)  and  multimedia  (i.e.,  CD-ROM 
players  and  production,  scanners,  electronic  classroom  components)  requirements.  Descnbe  the 
current  method  and  evaluate  the  operational  impact  of  maintaining  the  status  quo. 

2.  Proposed  Solution.  Provide  detailed  configuration  information.  Discuss  special 
requirements,  such  as  data  communications,  multimedia,  user  training,  site  preparation,  LAN 
installation  and  administration.  Identify  assumptions  and  constraints  considered  in  the  selection. 
Indicate  milestone  schedule  of  planned  events. 

3.  Other  Alternatives  Considered.  Summarize  the  alternatives  not  selected  and  explain 
the  shortcomings  based  on  technical  and/or  operational  feasibility,  and/or  cost  effectiveness. 

4.  Costs  and  Benefits.  Summarize  the  projected  costs  for  non-recurring  acquisition  and 
recurring  life  cycle  operations.  Identify  the  expected  benefits,  including  increased  productivity, 
improvements  in  function,  cost  avoidance,  and  cost  savings/offsets. 

5.  Interface  Considerations.  Describe  existing,  planned  or  potential  interfaces  with  sys- 
tems/procedures  external  and  internal  to  the  organization. 

6.  Testing.  Describe  the  applicable  developmental  and  operational  tests  to  be  conducted 
prior  to  implementation. 

7.  Funding.  Identify  funding  types,  sources,  and  availability. 

8.  Other  Comments.  Address  additional  requirements  and  considerations,  such  as  user 
training,  security,  site  preparation,  system  administration,  hardware/software  maintenance. 

The  requirements  to  document  life  cycle  support  issues,  planning  considerations,  costs  and 
benefits  in  an  ASDP  are  applicable  to  the  development  of  a  methodology  to  analyze  the  costs  and 
potential  benefits  of  converting  conventional  courses  or  computer-based  training  to  the  lETM/AC 
technology.  As  part  of  that  analysis,  a  variety  of  factors  that  impact  both  the  costs  and  the  benefits 
should  be  examined.  These  factors  include  the  characteristics  of  the  course,  the  current 
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requirements  for  course  management  versus  the  requirements  for  the  new  technology,  the  interface 
with  and  support  of  the  schoolhouse  infrastructure,  as  well  as  the  alternative  hardware  and  software 
capabilities  for  authoring  and  delivering  the  BETM/AC  instruction. 

2.3  Methodology 

Figure  2  provides  an  overview  of  the  process  used  to  analyze  the  costs  and  benefits  of 
implementing  the  lETM/AC  technology.  The  process  is  organized  into  seven  major  activities.  The 
relationship  between  each  of  the  activities  and  the  relevant  sections  of  an  ASDP  is  illustrated  in  the 
figure.  Although  the  activities  appear  discrete,  they  are  interdependent  and  somewhat  concurrent 
as  indicated  by  the  multiple  references  to  sections  in  an  ASDP.  The  following  paragraphs  briefly 
summarize  the  activities  and  the  inputs  to  an  ASDP.  A  detailed  description  of  each  of  these 
activities  is  provided  in  the  Section  3. 

The  initial  step  in  the  process  should  be  an  analysis  of  the  course  characteristics.  The 
potential  benefits  to  be  gained  by  automating  the  course  should  be  examined  to  determine  if  the 
course  is  a  good  candidate  for  conversion.  If  the  analysis  indicates  yes,  then  the  results  should  be 
useful  in  preparing  the  statement  of  the  need  (Section  1  of  an  ASDP)  and  a  preliminary  definition 
of  the  benefits  (Section  4  of  an  ASDP). 

If  the  potential  benefits  suggest  that  a  course  should  be  selected  for  conversion,  then  an 
analysis  of  the  requirements  for  managing  the  course  with  the  new  technology  versus  the  status  quo 
should  be  conducted.  The  analysis  should  consider  trade-offs  in  personnel  time,  personnel 
requirements,  facilities,  distribution  costs,  as  well  as  other  considerations.  As  shown  in  Figure  2, 
the  analysis  of  course  management  requirements  should  provide  inputs  for  preparing  the  statement 
of  the  need  (Section  1  of  an  ASDP),  a  preliminary  set  of  requirements  for  the  proposed  solution 
(Section  2  of  an  ASDP),  and  a  statement  of  other  special  requirements  (Section  8  of  an  ASDP). 

If  the  trade-offs  in  course  management  requirements  indicate  that  the  course  should  be 
converted,  then  an  analysis  of  the  requirements  and  costs  associated  with  course  conversion  should 
be  conducted.  This  analysis  should  focus  on  redesigning  the  course  to  take  advantage  of  the 
enhanced  instructional  capabilities  of  the  lETM/AC  technology.  The  capabilities  required  for 
authoring  and  multimedia  development  and  support  should  be  explored  in  terms  of  personnel 
resources  and  costs.  The  results  of  this  analysis  should  provide  inputs  for  preparing  the  statement 
of  the  proposed  solution  (Section  2  of  an  ASDP),  the  summary  of  costs  (Section  4  of  an  ASDP), 
and  description  of  the  interface  requirements  (Section  5  of  an  ASDP). 

The  analysis  of  delivery  system  capabilities  should  include  instructor  and  student  station 
hardware  and  software  configurations,  other  automated  classroom  components,  networking 
requirements,  and  classroom  furniture.  The  capabilities  of  different  configurations  and  various 
options  should  be  explored  thoroughly  and  the  costs  should  be  compared  to  identify  the  most  cost 
effective  solution.  As  indicated  in  Figure  2,  the  results  of  the  analysis  should  be  used  in  preparing 
a  detailed  statement  of  the  proposed  solution  (Section  2  of  an  ASDP)  and  a  discussion  of  the  other 
alternatives  that  were  investigated  (Section  3  of  an  ASDP). 
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Cost/Benefits  Analysis  Activities 


Relevant  ASDP  Sections 


Analysis  of  Course  Characteristics  and 
Potential  Benefits  to  Determine  if  Course 
Should  be  Converted 


Input  to  ASDP  - 
Sections  1  &  4 


Analysis  of  Trade-offs  in  Course/School 
Management  Requirement  to  Determine 
if  Course  Should  be  Converted 


Input  to  ASDP  - 
Sections  1, 2,  &  8 


- 

Analysis  of  Requirements  to  Convert 

Input  to  ASDP  - 

Course  to  lETM/AC  Technology 

- ^  Sections  2, 4,  &  5 

Analysis  of  Delivery  System  Capabilities 
to  Determine  Alternative  Configurations 


Input  to  ASDP  - 
Sections  2  &  3 


Analysis  of  Interface  Considerations 
wiA  Infrastructure  to  Determine 
Configuration 


Input  to  ASDP  - 
Sections  2,  3, 4, 
5,6,&8 


Development  of  Non-Recurring  and 
Recurring  Cost  Estimates  for 
Implementation  of  lETM/AC 


Input  to  ASDP 
Section  4 


Identification  of  Quantifiable  and 
Nonquantificable  Benefits  for 
Implementation  of  lETM/AC 


Input  to  ASDP 
Section  4 


Figure  2.  Costs/benefits  analysis  process. 
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After  the  previous  activities  have  been  accomplished,  an  analysis  of  the  interfaces  with  the 
schoolhouse  infrastructure  should  be  conducted  to  determine  configuration  and  support 
requirements.  The  analysis  should  consider  factors  such  as  site  preparation,  special  facilities 
requirements,  communications,  networking,  multimedia  development  capabilities,  CD-ROM 
mastering  and  production,  software  inventory  and  licensing  control,  customized  software  support, 
hardware  maintenance  support,  life  cycle  support,  as  well  as  lETM  database  maintenance 
functions.  The  results  of  the  analysis  should  provide  inputs  for  preparing  the  statement  of  the 
proposed  solution  (Section  2  of  an  ASDP),  a  discussion  of  the  alternative  configurations  considered 
(Section  3  of  an  ASDP),  the  summary  of  costs  (Section  4  of  an  ASDP),  a  description  of  the 
interface  requirements  (Section  5  of  an  ASDP),  a  plan  for  testing  (Section  6  of  an  ASDP),  and  a 
statement  of  other  special  requirements  (Section  8  of  an  ASDP). 

Non-recurring  (one  time)  and  recurring  (annual)  cost  estimates  should  be  developed  for  the 
proposed  solution  and  any  other  alternatives  that  are  considered  feasible.  As  shown  in  Figure  2,  the 
results  of  this  activity  should  be  the  summary  of  projected  costs  (Section  4  of  an  ASDP). 

The  potential  benefits  that  were  identified  during  the  initial  analysis  should  be  evaluated 
further.  These  benefits  should  be  quantified,  where  possible,  to  provide  justification  for  the 
proposed  solution.  The  results  of  this  activity  should  be  the  statement  of  benefits  (Section  4  of  an 
ASDP). 


15 


Section  Three 


Costs/Benefits  Analysis  Process 


3.0  Costs/Benefits  Analysis  Process 


3.1  Overview 

The  process  used  to  analyze  the  costs  and  benefits  of  implementing  the  lETM/AC 
technology  in  NAVEDTRACOM  schools  was  summarized  in  Section  2.3.  Each  of  the  seven 
activities  listed  below  is  described  in  detail  in  this  section. 

1.  An  analysis  of  the  course  characteristics  and  the  potential  benefits  to  determine  if  the 
course  should  be  converted. 

2.  An  analysis  of  the  trade-offs  in  course/school  management  requirements  to  determine 
if  the  course  should  be  converted. 

3.  An  analysis  of  the  requirements  to  convert  the  course  to  the  lETM/AC  technology. 

4.  An  analysis  of  delivery  system  capabilities  to  determine  alternative  configurations. 

5.  An  analysis  of  the  interface  considerations  with  the  infrastructure  to  determine  the 
configuration  and  support  requirements. 

6.  The  development  of  non-recurring  and  recurring  cost  estimates. 

7.  The  identification  of  quantifiable  and  nonquantifiable  benefits. 

3.2  Analysis  of  Course  Characteristics 

The  characteristics  of  a  course  or  courses  within  a  school  need  to  be  analyzed  to  determine 
the  potential  benefits  that  may  be  realized  by  introducing  the  lETM/AC  technology.  The  analysis 
of  course  characteristics  may  encompass  conventional  paper-based  group-paced  instruction  as  well 
as  computer-based  self-paced  instruction  that  could  benefit  from  incorporation  of  an  lETM 
database  or  automated  classroom  components.  Implementing  several  courses  may  be  an  important 
consideration  in  achieving  cost  savings  and  other  efficiencies. 

Table  1  is  designed  to  provide  a  framework  for  the  analysis.  In  the  left-hand  column  of  the 
table,  the  course  characteristics  are  organized  into  broad  categories  with  relevant  questions  posed 
for  each  category.  In  the  right-hand  column,  potential  benefits  that  relate  to  the  questions  are  stated. 
The  categories,  questions,  and  benefits  in  Table  1  are  representative  of  the  issues  that  need  to  be 
addressed.  They  are  not  meant  to  be  comprehensive  since  any  given  course  may  have  unique 
characteristics  that  impact  the  benefits. 

As  defined  in  the  table,  the  first  two  characteristics  in  the  analysis  of  potential  benefits  deal 
with  how  the  course(s)  fit  into  the  training  pipeline  and  what  student  throughput  is  projected. 
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Table  1 


Analysis  of  Course  Characteristics  and  Potential  Benefits 


Course  Characteristics  _ Potential  Benefits 


Training  Pipeline  and  Course  Length 

Where  is  the  course  taught  in  the  training  pipeline?  Is 
this  a  core  course  that  feeds  several  strands  in  the 
training  pipeline?  Can  the  training  pipeline  be 
shortened?  Can  die  content  be  included  in  another 
course  to  reduce  overall  training  time?  Can  the  use  of 
lETM/AC  technology  reduce  the  length  of  the  course? 
With  lETMs  in  use  in  the  fleet,  can  proceduralized 
portions  of  the  course  be  designed  as  just-in-time 
training  or  job  performance  aids? 

Reduced  Student  Training  Costs 

By  eliminating  a  course  or  reducing  the  length  of  a 
course  it  may  be  possible  to  shorten  the  overall  training 
pipeline.  A  reduction  in  training  time  translates  to 
student  cost  savings.  A  reduction  in  training  time  may 
also  translate  to  student  TDY  cost  savings.  ETM-based 
just-in-time  training  or  job  performance  aids  may 
enhance  fleet  operational  capability,  while  reducing 
formal  school  training  time  and  costs. 

Student  Throughput 

What  is  the  student  throughput?  Is  the  throughput 
projected  to  increase  or  decrease  in  the  future?  Are 
there  plans  to  cranbine  related  technician  ratings? 

Lower  Cost  per  Student  per  Course 

The  higher  the  student  throughput,  the  lower  the  cost 
per  student  per  course. 

Academic  Attrition/Setback  Rate 

What  is  the  academic  attrition/setback  rate?  How  much 
mandatory  study  time  is  required? 

Reduced  Student  Trainii^  Costs 

The  ETM/AC  technology  may  reduce  academic 
attrition,  thereby  avoiding  student  training  costs. 

Student  Proficiency 

Are  student  performance  scores  adequate?  Can  the 
scores  be  improved?  Is  more  proficiency  required  in  the 
fleet? 

Improved  Student  Performance  Scores 

The  technology  may  improve  student  performance 
scores  and  increase  proficiency,  while  reducing  student 
training  time. 

Format  of  Course  Materials 

What  percentage  of  the  materials  are  in  electronic 
format?  What  percentage  are  in  paper  format  only? 

Reduced  Development  and  Revision  Time 

Electronic  formats  reduce  the  time  to  develop,  store, 
distribute,  and  revise  training  materials. 

Stability  of  Course  Content 

How  often  does  the  technical  content  of  the  course 
change?  How  significant  are  the  revisions?  How 
important  is  it  to  have  current,  accurate  training?  Does 
the  course  use  the  technical  manuals/documentation 
that  is  revised  frequently?  How  time-consuming  is  it  to 
revise  the  IG,  TG,  and  tests?  Are  information  sheets 
included  in  the  TGs  along  with  assigmnent  sheets?  How 
long  does  it  take  instructors  to  personalize  the  revised 
IG? 

Reduced  Revision  T  ime 

If  the  technical  course  content  must  be  changed 
regularly  to  present  current,  accurate  information,  then 
the  course  should  be  developed  in  electronic  format.  It 
requires  less  time  to  revise  IGs,  TGs,  tests,  and  other 
supporting  instructional  materials  in  electronic  format. 
Dissemination  of  up-to-date  technical  documentation 
on  CD-ROM  significanUy  reduces  revision  time. 
Reduced  Instructor  Personalization  Time 

After  course  revision,  instructor  personalization  time  is 
reduced  by  making  annotations  in  an  electronic  IG. 
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Table  1  (Continued) 


Course  Characteristics  Potential  Benefits 


Course  GraphicsA^isuals 

Are  graphics/visuals  an  important  aspect  of  the  course? 
How  many  gr^hics/visuals  are  used  in  the  course?  Are 
more  graphics/visuals  needed?  What  quality  are  the 
graphics/visuals?  Are  they  paper-based  or  in  electronic 
format?  Does  the  instructor  or  a  graphics  artist  develop 
the  visuals?  Is  a  word  processing  draw  package  used? 
Does  the  course  present  concepts  that  are  difficult  to 
grasp  using  traditional  media  and  graphics?  For 
example,  can  animations  illustrate  concepts  more 
effectively? 

Reduced  Development  and  Revision  Time 

Depending  on  the  complexity  of  the  visuals,  creating 
digitized  graphics  may  increase  or  decrease  the  initial 
development  time,  but  the  revision  time  can  be 
significantly  reduced.  Scanning  paper-based  graphics 
may  result  in  degradation  of  image  and  usability, 
especially  if  the  print  quality  is  poor  or  the  graphic  is  a 
fold-out. 

Improved  Student  Performance  Scores 

After  course  revision,  instructor  personalization  time  is 
reduced  by  making  annotations  in  an  electronic  IG. 

Course  Media/Materials 

What  media/material  are  used  in  the  course?  Is  it 
lecture-based  with  chalk  talk?  Are  wall  charts, 
transparencies,  slides,  audio/visual  aids,  mock-ups 
being  used?  Is  the  course  structured  to  include  both 
classroom  and  laboratory  sessions?  What  kind  of  lab 
sessions  are  conducted?  How  much  time  is  spent  in  the 
lab?  Can  lab  exercises  be  simulated  in  the  classroom  to 
alleviate  bottlenecks  in  the  lab?  Are  technical  manuals 
used  in  the  classroom  and/or  lab?  Has  an  lETM 
database  already  been  developed?  Can  other  media  be 
introduced,  such  as  interactive  courseware  (ICW)  to 
facilitate  learning?  Can  portions  of  the  course  be  self- 
paced  rather  than  group-paced  instruction?  Can 
students  work  together  in  groups? 

Reduced  Development  and  Revision  Time 

By  eliminating  a  course  or  reducing  the  length  of  a 
course  it  may  be  possible  to  shorten  the 

Reduced  Management/Distribution  Costs 

Electronic  format  reduces  the  costs  for  course  supplies, 
media/materials  inventory  and  storage  space, 
reproduction  services,  training  materials  distribution, 
lETMs  facilitate  the  use  of  up-to-date,  current  technical 
manuals  in  the  classroom. 

Improved  Student  Performance  Scores 

The  technology  may  facilitate  student  learning  and 
increase  proficiency. 
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The  next  two  characteristics  in  Table  1  address  student  performance  and  proficiency.  The 
last  four  characteristics  focus  on  course  design,  format,  content,  and  media/materials. 

The  most  significant  benefits  to  be  realized  are  reductions  in  student  training  costs.  This  is 
particularly  true  if  these  reductions  are  accompanied  by  better  trained  personnel  returning  to  the 
fleet  in  a  shorter  period  of  time.  Another  important  benefit  is  an  increase  in  student  proficiency  that 
translates  to  improved  operational  readiness.  One  consideration  for  implementing  the  lETM/AC 
technology  in  flie  schoolhouse  may  be  not  only  to  train  technicians  to  perform  operation  and 
maintenance  tasks,  but  to  also  provide  training  on  how  to  use  the  lETMs  that  are  being  introduced 
into  the  fleet.  Other  potential  benefits  for  introducing  the  new  technology  may  be  the  capability  to 
deliver  more  up-to-date,  technically  accurate,  and  effective  instruction  in  conjunction  with 
reductions  in  course  development,  management,  distribution,  and  revision  time. 

The  results  of  this  analysis  provide  inputs  for  preparing  an  ASDP.  Examining  the  course 
characteristics  and  identifying  potential  benefits  helps  establish  the  need  for  automating  the  course 
(Section  1)  and  provides  a  preliminary  statement  of  the  benefits  (Section  4). 

3.3  Analysis  of  Trade-offs  in  Course  Management  Requirements 

The  analysis  of  the  requirements  to  manage  paper-based  courses  versus  lETM/ AC-based 
courses  is  a  complex  undertaking  that  should  be  addressed  at  the  school  level.  Implementation  of 
the  new  technology  in  just  one  course  in  one  classroom  in  a  schoolhouse  can  have  consequences 
that  extend  beyond  that  classroom.  For  the  most  part,  the  management  requirements  for  the  two 
alternatives  are  very  different  and  have  implications  for;  (1)  the  organization  and  its  structure, 
policies,  and  procedures;  (2)  the  instructors  and  their  duties;  (3)  the  roles  and  responsibilities  of 
management,  administrative,  and  support  personnel;  (4)  the  facilities  and  their  utilization;  and  (5) 
funding  for  life  cycle  support  of  the  automated  classroom  components.  If  the  status  quo  includes 
management  of  computer-based  training  in  a  classroom  setting,  then  the  analysis  wiU  only  need  to 
take  into  account  additions  and/or  modifications  in  requirements  as  a  result  of  the  new  technology. 

The  analysis  must  look  at  the  trade-offs  between  the  alternatives  to  determine  if  the 
potential  benefits  identified  are  offset  by  the  changes  in  management  requirements.  In  some  cases, 
the  changes  in  management  requirements  may  simplify  procedures,  reduce  time  and  effort,  or 
make  more  efficient  use  of  facility  space.  In  other  cases,  the  changes  may  add  a  number  of  new 
functions  to  be  performed  and  requirements  to  be  supported,  require  personnel  training  for  new 
skills,  dictate  a  different  mix  of  personnel,  or  increase  time  and  effort.  All  of  these  changes  have 
cost  implications  that  must  be  addressed  as  part  of  the  decision  making  and  planning  process.  With 
the  rapid  evolution  in  technologies  and  advances  in  system  capabilities,  the  analysis  of 
management  requirements  should  be  viewed  as  an  ongoing  process  both  during  and  after 
implementation. 

This  analysis  results  in  inputs  for  preparing  an  ASDP.  The  evaluation  of  the  requirements 
for  managing  the  status  quo  course(s)  versus  the  converted  course(s)  should  provide  clarification 
of  the  need  to  implement  the  lETM/AC  technology  (Section  1).  The  analysis  of  trade-offs  should 
help  define  the  best  approach  (Section  2)  and  identify  additional  requirements  that  must  be 
addressed  (Section  8). 


22 


Table  2  is  designed  to  assist  school  personnel  in  conducting  the  analysis.  The  left-hand 
column  of  the  table  provides  general  definitions  of  course  management  requirements.  There  may 
be  other  requirements  and  considerations  for  any  given  course.  The  list  of  general  requirements  is 
meant  to  be  representative  and  serve  as  guidelines  for  specific  applications.  The  center  column 
describes  the  implications  for  meeting  that  requirement  in  a  traditional  paper-based  course,  while 
the  right-hand  column  summarizes  the  implications  in  an  lETM/AC-based  course. 

The  first  four  requirements  in  Table  2  relate  to  instructors  and  their  roles  in  and  out  of  the 
classroom.  Implementation  of  the  new  technology  has  a  profound  effect  on  the  duties  that  the 
instructors  perform  and  the  skills  that  instructors  require.  One  of  the  key  requirements  is  to  provide 
training  on  how  to  use  the  technology  to  develop,  personalize,  and  deliver  instruction.  Although 
some  members  of  the  instructional  staff  may  be  computer  literate,  some  type  of  formal  instructor 
training  is  necessary  to  bring  all  of  the  staff  up  to  at  least  a  minimum  level  of  competency  and 
reduce  the  learning  curve  time.  The  formal  instructor  training  needs  to  include  such  features  as 
hardware/software  setup  and  operations,  techniques  for  training  students  to  use  the  system,  making 
annotations  to  personalize  the  instruction,  developing  on-line  training  aids  using  computer-based 
graphics  software,  using  the  system  capabilities  for  monitoring  student  activities,  controlling 
classroom  activities  and  time,  and  performing  automated  testing  functions. 

The  next  two  requirements  in  the  table  relate  to  classroom  facilities  and  scheduling.  The 
trade-offs  here  are  dependent  upon  the  adequacy  and  availability  of  classrooms  in  the  schoolhouse. 
The  implementation  of  the  lETM/AC  technology  may  require  extensive  classroom  refurbishment 
and  costly  preparation.  However,  these  costs  can  be  offset  if  several  courses  can  share  the 
classroom.  Scheduling  several  courses  in  an  automated  classroom  may  require  changes  in  shifts, 
create  problems  of  shared  access,  and  introduce  security  issues. 

The  requirements  to  maintain,  reproduce,  and  distribute  course  materials  in  Table  2  are 
interrelated.  These  trade-offs  in  course  management  requirements  favor  using  the  new  technology. 
For  the  most  part,  having  materials  in  electronic  format  simplifies  the  procedures,  decreases  the 
inventory  space  that  is  required,  and  reduces  costs  as  well  as  personnel  time.  However,  the 
advantages  may  be  negated  by  the  need  to  perform  system  administration  functions  for  the 
automated  classrooms  that  are  not  required  in  conventional  classrooms.  Two  other  requirements 
shown  in  the  table,  networking  and  system  administration,  go  hand-in-hand.  The  decision  to  use  a 
local  area  network  (LAN)  in  the  classroom  has  an  impact  on  the  system  administration  functions 
that  need  to  be  performed. 

The  requirement  for  personnel  training  is  often  overlooked  when  implementing  new 
technology.  In-house  management  and  support  personnel  may  require  specialized  training  to 
handle  the  network  and  system  administration  requirements.  The  type  and  amount  of  training  and 
the  respective  costs  are  a  function  of  the  mix  of  personnel  skills  and  capabilities.  Training  may  not 
be  required,  however,  if  the  decision  is  made  to  use  contractor  support  to  perform  LAN  and  system 
administration  functions. 
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Table  2 


Analysis  of  Course/Schoolhouse  Management  Requirements 


Definition  of  Requirement 


Instructor  Training 
Formal  instructor  training  addresses 
course  design,  development,  and 
implementation  with  emphasis  on 
group-paced  instructor-led  training. 


Instructor  Preparation 
The  time  an  instructor  spends 
preparing  for  classroom  sessions 
may  vary  depending  on  the 
individual  instructor’s  experience 
level  and  the  stability/maturity  of  the 
course.  _ _ _ 


Instructor  Personafization 
Individual  instructors  make  personal 
annotations  in  the  Related 
Instructional  Activity  (RIA)  column 
inIG 


Instructor  Classroom 
Management 

Instructors  deliver  instruction, 
manage  classroom  time,  interact 
with  students,  monitor  student 
progress,  administer  tests  and 
remediation. 


Classroom  Facilities 
The  physical  classroom  space 
including  furniture,  training  media 
devices,  equipment,  utilities,  etc. 
The  availability  and  adequacy  of 
classroom  facilities  varies  from 
school  to  school. 


Paper-based  Course 


Instructors  are  subject  matter  experts 
who  have  been  trained  to  design, 
develop,  and  deliver  platform 
instruction  with  traditional 
instructional  media/materials. 


Delivering  training  with  traditional 
instructional  media/materials  (such 
as  paper-based  IG  and  TGs, 
chalkboards,  transparencies,  tech 
manuals  requires  a  minimal  amount 
of  instructor  preparation  time. 


Depending  on  course  complexity, 
frequency  of  revisions,  and 
instructor’s  experience  with 
content,  instructor  personalization 
may  be  a  time-consuming  activity. 

Delivering  training  with  paper-based 
IG,  Tgs,  and  technical  manuals  can 
make  it  difficult  for  an  instmctor  to 
monitor  individual  student  progress. 
Instructors  must  interact  with 
students  by  asking  questions  to 
determine  their  progress.  Classroom 
time  may  be  wasted. 


Classroom  facilities  for  a  paper- 
based  course  with  conventional 
instructional  media/material  may  be 
used  for  a  number  of  courses  or  for  a 
limited  number  of  courses  if  specific 
equipment  or  extensive  sets  of 
technical  manuals/documentation  is 
required.  Classroom  facilities  do 
not  require  special  electrical  wiring, 
air  conditioning,  flooring,  etc. 


lETM/AC-based  Course 


Instructors  require  training  to  use  the 
technology  for  instructional 
development  and  classroom 
presentation.  Approximately  40 
hours  of  initial  training,  depending 
on  the  hardware  and  software,  plus 
some  follow-on  sessions  may  be 
required. 


Use  of  the  new  technology  requires  | 
an  extensive  amount  of  instructor 
preparation  time  and  new  tasks  like 
checking  set,  up  of  the  student  and 
instructor  stations.  As  instructors  get 
up  on  the  learning  curve,  this  time 
should  decrease.  _ _ 


Capability  for  instructors  to  make  a 
standard  set  of  annotations  for  new 
or  revised  courses,  share  personal 
annotations  on-line,  use  annotations 
to  link/launch  to  other  media  objects 
such  as  BETMs. _ _ 


Use  of  the  new  technology  allows 

the  instructor  to  control  the 
presentation  and  monitor  the  class  or 
individual  student  progress 
simultaneously.  Use  of  classroom 
I  time  is  more  efficient  with  on-line 
rather  than  paper-based  training 
niaterials  and  technical  manuals. _ 


Classroom  facilities  for  an  lETM/ 
AC-based  course  may  be  used  for  a 
number  of  courses.  The  only 
requirement  is  to  have  the 
appropriate  course  materials 
available  on  the  instructor  and 
student  stations.  Classroom  facilities 
require  electrical  wiring,  cabling,  air 
conditioning,  flooring,  etc.  to 
support  the  hardware. 
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Table  2  (Continued) 


Definition  of  Requirement 

Paper-based  Course 

lETM/AC-based  Course 

Classroom  Scheduling 

Classroom  scheduling  is  a  function 
of  the  student  throughput,  course 
convene  dates,  and  the  availability 
and  adequacy  of  classroom 
facilities. 

Generic  classroom  facilities  simplify 
course  scheduling.  During  peak 
loads,  scheduling  classrooms 
designed  for  specific  courses  may  be 
difficult  and  require  extra  shifts. 

Scheduling  is  a  major  concern  for 
lETM/AC  facilities.  When  the 
number  of  classrooms  is  limited, 
scheduling  must  consider  multiple 
classes  as  well  as  study  time.  The 
^propriate  course  materials  must  be 
accessible  at  the  correct  time. 

Course  Inventory 

An  inventory  of  all  of  the  media/ 
materials  required  to  present  a 
course  must  be  maintained  in  a 
secure  space.  These  items  include 
course  consumables,  media  delivery 
devices  and  spare  components, 
multiple  sets  of  technical  manuals, 
training  aids,  instructional  materials 
and  tests. 

For  traditional  paper-based  courses, 
an  adequate  inventory  of  course 
consumable  items,  media  delivery 
device  components  (i.e.,  overhead 
projectors),  training  aids,  training 
materials  (IGs,  TGs,  tests),  and  sets 
of  technical  manuals  for  students  to 
use  in  class  must  be  maintained. 

For  lETM/AC-based  courses,  most 
of  the  items  in  the  course  inventory 
are  sttwed  in  electronic  format  (i.e., 
CD-ROM,  hard  disks,  or  floppy 
disks).  An  inventory  of  consumables 
and  media  delivery  device 
components  (i.e.,  spare  mouse,  light 
bulbs,  headphones)  is  still  required, 
but  the  facility  space  is  significantly 
reduced. 

Course  Reproduction  Costs 

The  costs  for  reproducing  the  course 
training  materials,  including  the  IGs, 
TGs,  tests,  training  aids,  other 
training/reference  materials,  and  sets 
of  technical  manuals. 

The  lead  time  and  the  costs  for 
reproducing  multiple  copies  of 
p^r-based  course  materials  and 
transparencies/slides  at  the  base 
printing  ship/support  office  may  be 
significant  depending  on  the  course 
content  and  structure. 

The  costs  as  well  as  the  time 
required  to  generate  multiple  copies 
of  course  materials  in  electronic 
format  is  significantly  reduced. 

Course  Distribution 

Procedures  for  establishing 
accountability  and  maintaining 
security  in  the  distribution  of 
instructional  materials  and  tests  for 
each  course  convene. 

The  effort  to  distribute  the  needed 

• 

paper  copies  of  instructional 
materials  and  tests  may  be 
significant,  especially  if  each  student 
requires  a  set  of  technic^  manuals. 
Instructors  are  responsible  for 
security  of  the  materials. 

The  effort  to  distribute  the  necessary 
instructional  materials  and  tests  in 
electronic  format  is  significantly 
reduced.  However,  the  problems 
with  establishing  accountability  and 
maintaining  security  are  increased. 

Networking 

Use  of  a  local  area  network  (LAN) 
to  establish  a  client/server 
relationship  between  the  instructor 
and  student  stations  in  a  classroom. 

As  defined,  networking  is  not  a 
feature  of  a  traditional  paper-based 
course. 

In  most  cases,  networking  is  an 
important  aspect  of  efficiently 
managing  lETM/AC  courses.  A 
trained  LAN  administrator  is 
required  to  support  the 
implementation. 
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Table  2  (Continued) 


Definition  of  Requirement 

Paper-based  Course 

lETM/AC-based  Course 

System  Administration 

Functions  include,  but  are  not 
limited  to,  providing  assistance  to 
instructor  and  staff,  managing 
software  inventory,  establishing 
configuration  control,  performing 
backups,  performing  preventative 
maintenance  and  diagnostics, 
interfacing  with  vendors  and  support 
activities,  establishing  procedures 
and  policies. 

As  defined,  system  administration 
functions  are  not  necessary  for 
managing  paper-based  courses. 

System  administration  functions  are 
essential  for  managing  lETM/AC- 
based  courses.  The  number  of 
administrators  required  depends  on 
the  capabilities  of  the  personnel,  the 
number  of  student  and  instructor 
stations,  the  number  of  courses,  the 
complexity  of  the  configuration,  the 
use  of  networks,  security  issues,  the 
types  of  support  that  are  available 
in-house  or  by  contractors. 

Personnel  Ti'aining 

Course  management  and  support 
personnel  may  require  specific 
training  to  perform  their  functions. 

For  traditional  paper-based  courses, 
no  training  or  minimal  training  may 
be  required.  For  the  most  part,  no 
specialized  skills  or  knowledge  are 
necessary. 

The  system  administration 
functions  introduced  by  the  ITEM/ 
AC  technology  require  specialized 
training  for  management  and 
support  personnel. 

Performance  Testing 

Student  testing  during  the  course 
and  performance  assessment  at  the 
end  of  the  course. 

For  traditional  paper-based  courses, 
the  Instmctional  Support  System/ 
Micro  Standing  Training  Activity 
Support  System  (ISS/MicroSTASS) 
requires  hand  scanning  of  multiple 
choice  answer  sheets  for  exams. 
Analysis  and  reporting  is  performed 
at  central  location. 

The  Audience  Tailored  Instructional 
Testing  Environment  (ATITE) 
allows  on-site  rapid  analysis  of 
student  scores  immediately  after 
tests.  On-line  testing  reduces  paper 
requirements  and  allows  instructors 
to  generate  any  number  of  random 
tests  from  test  item  database. 

Remediation  and  Study 

Instructors  need  to  diagnose  student 
problems  and  prescribe  remediation 
activities  and  outside  study. 

For  traditional  paper-based  courses, 
students  can  study  classroom  notes 
and  complete  remediation  activities 
anytime  and  anywhere. 

For  lETM-based  courses,  smdents 
must  have  access  to  the  lETM 
systems.  This  adds  to  demands  on 
classroom  time,  unless  separate 
study/remediation  systems  are  made 
available  to  students. 
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The  implications  for  testing  and  remediation  should  also  be  considered.  Automated 
performance  assessment  and  record  keeping  software  may  provide  instructors  with  a  variety  of 
features  that  are  not  available  in  paper-based  testing.  However,  to  achieve  maximum  effectiveness 
and  efficiency,  these  features  need  to  be  integrated  with  the  schoolhouse  performance  reporting 
requirements.  Automated  performance  assessment  software  may  also  allow  instructors  to  diagnose 
specific  remediation  activities  for  students  having  problems.  But,  scheduling  remedial  training 
sessions  needs  to  consider  constraints  on  automated  classroom  resources. 

3.4  Analysis  of  Course  Conversion  Requirements 

The  introduction  of  lETM/AC  technology  offers  new  vistas  for  instructional  development 
and  delivery.  The  capability  to  present  instruction  using  interactive  graphics,  animations,  and 
immediate  access  to  up-to-date  technical  information  provides  new  design  possibilities.  Most,  if 
not  all,  paper-based  courses  that  are  converted  to  include  an  on-line  lETM  database  should  be 
redesigned  to  take  advantage  of  the  enhanced  capabilities  that  are  available.  Just  presenting  lETM 
references  electronically  in  the  classroom  rather  than  physically  searching  the  technical  manuals  is 
not  utilizing  the  technology  to  its  fullest  extent. 

In  current  practice,  the  process  of  developing  an  lETM  to  be  used  in  training  and  in  the  fleet 
is  separate  from  the  process  of  developing  instructional  materials  for  use  in  the  formal  school.  A 
discussion  of  the  options  and  processes  for  authoring  lETMs  and  the  relationship  of  the  processes 
to  authoring  instructional  materials  is  beyond  the  scope  of  this  study.  However,  they  are  discussed 
in  detail  in  a  survey  of  lETM  practices  in  military,  commercial  and  higher  education  applications 
that  was  conducted  under  contract  to  NRPDC  (Kribs  &  Mark,  1995). 

Whether  training  materials  are  going  to  be  integrated  with  the  lETM  database  and  how  the 
lETM  elements  are  going  to  be  launched  in  the  training  materials  will  impact  the  course  conversion 
effort.  The  requirements  for  converting  the  course  should  be  based  on  an  analysis  of  the  links 
between  the  training  materials  and  the  electronic  technical  documentation  and  the  tools  and 
processes  to  accomplish  these  links. 

The  number  of  technical  manuals  to  be  included  in  the  lETM  database  as  well  as  the 
accuracy,  readability,  and  legibility  of  their  content  will  also  be  a  factor  in  the  course  conversion 
effort.  For  a  mature  operational  system,  the  quality  of  the  legacy  data  that  is  scanned  or  digitized 
into  electronic  format  may  not  be  adequate  for  classroom  presentation.  Fold-out  schematics  that 
have  been  converted  as  separate  pages  in  an  electronic  format  or  detailed  illustrations  that  have 
been  raster  scanned  may  not  be  of  sufficient  resolution  and  clarity  for  use  as  training  media  in  an 
automated  classroom. 

3.4.1  Conversion  Process 

There  are  two  sets  of  procedures  for  developing  and  revising  curriculum  in  the  Navy.  The 
Personnel  Performance  Profiles  (PPP)  based  procedures  are  documented  in  the  NAVEDTRA  131 
training  manual  series  and  the  task  based  procedures  are  documented  in  the  NAVEDTRA  130 
training  manual  series.  The  process  for  converting  a  traditional  course  to  the  DETM/AC  format 
should  follow  the  guidelines  for  revising  an  existing  course  that  are  documented  in  the  appropriate 
training  manual  series. 
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The  introduction  of  the  lETM/AC  technology  may  result  in  changes  in  the  learning 
objectives  and  the  instructional  sequence  in  both  PPP-based  and  task-based  courses.  It  may  also 
result  in  changes  in  student  testing  and  performance  assessment  techniques.  For  example,  paper- 
based  short  answer  tests  may  be  replaced  by  on-line  multiple  choice  test  questions  randomly 
selected  from  a  test  item  pool.  The  capabilities  of  performance  assessment  and  management 
software  packages  to  provide  on-line  testing  and  reduce  administrative  record  keeping  tasks  should 
be  investigated  for  use  in  the  automated  classroom. 

The  lETM/AC  format  will  impact  the  Resource  Requirements  List  developed  for  both 
types  of  courses.  A  survey  of  commercial-off-the-shelf  (COTS)  software  should  be  conducted  to 
see  what  capabilities  are  available  at  what  cost  as  part  of  developing  the  Resource  Requirements 
List  for  the  revised  course.  Any  number  of  COTS  packages  may  provide  inexpensive,  easy-to-use 
features  to  augment  deficiencies  in  the  visuals  in  the  lETM  database  and  enhance  the  effectiveness 
of  the  instruction. 

As  mentioned  earlier,  the  Navy’s  AIM  (Authoring  Instructional  Materials)  software  is  used 
to  author  PPP-based  lesson  plans  and  trainee  guides,  while  standard  word  processing  software 
(Word  Perfect)  is  currently  used  to  author  task-based  lesson  plans  and  trainee  guides.  Regardless 
of  the  software  used  to  convert  the  course  to  the  lETM/AC  format,  there  must  be  a  capability  to 
link  and  launch  to  other  media  including  lETMs,  interactive  courseware  (ICW),  video,  audio, 
animations,  and  graphics. 

The  format  neutral  architecture  provided  by  the  Training  Integration  Management  Software 
(TIMS)  should  enhance  the  capability  to  integrate  instructional  materials  from  all  Windows-based 
sources  into  the  lETM/AC  format.  Tliere  is  dso  an  ongoing  effort  by  the  Naval  Aviation  Warfare 
Center/Training  Systems  Division  (NAWC/TSD)  to  revise  AIM  to  operate  in  a  Windows-based 
environment  and  to  allow  authoring  of  both  task-based  as  well  as  PPP-based  instructional 
materials.  The  impact  of  these  new  versions  of  AIM  as  well  as  advances  in  electronic  publishing 
technology  for  SGML  and  HTML  (Hypertext  Markup  Language)  need  to  be  monitored. 

The  following  types  of  questions  concerning  instructional  development  support  should  be 
asked  as  part  of  the  conversion  process: 

•  Is  there  an  organization  that  provides  authoring  support  using  AIM  or  conventional 
word  processing  software?  How  are  requirements  integrated? 

•  What  COTS  software  is  required  to  support  the  redesign  of  the  instructional  materials? 

•  Are  there  requirements  for  COTS  or  customized  applications  software  (i.e.,  perfor¬ 
mance  assessment  and  management)? 

•  Is  there  an  organization  that  provides  in-house  or  contractor  support  developing  com¬ 
puter-based  training  materials  in  the  form  of  ICW,  animations,  enhanced  graphics  and 
visuals?  How  are  requirements  integrated?  What  are  the  costs? 

•  Are  there  requirements  for  the  \fisual  Information  Support  Center  to  provide  support? 
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•  Are  there  requirements  for  specialized  hardware  or  services  (i.e.,  conversion  to  vector 
graphics)?  If  so,  what  are  the  costs? 

3.4.2  Cost  Estimating  Parameters 

A  standard  metric  for  converting  conventional  courses  to  an  lETM/AC  format,  like  the 
metric  used  to  estimate  the  costs  for  developing  computer-based  training  (i.e.,  200  labor  hours  per 
hour  of  instruction),  cannot  be  developed  until  more  experience  is  gained.  The  parameters  for 
estimating  conversion  costs  must  rely  on  conventional  cost  estimating  techniques  for  revising  an 
existing  course.  That  is,  the  personnel  costs  for  performing  the  necessary  analysis,  design,  and 
lesson  development  activities,  including  linking  and  launching  to  media  objects,  must  be 
estimated.  The  costs  for  instructional  media  (i.e.,  graphics,  animations,  ICW,  audio,  video) 
development  to  take  advantage  of  the  automated  classroom  capabilities  and  enhance  the 
effectiveness  of  the  instruction  must  also  be  estimated.  Depending  on  the  organization,  the 
developers  may  be  instructors,  in-house  staff,  contractor  personnel  or  a  combination  of  the 
above. 


The  results  of  the  analysis  of  course  conversion  requirements  provides  inputs  into 
preparing  the  statement  of  the  proposed  alternative  (Section  2  of  the  ASDP),  the  summary  of  costs 
(Section  4  of  the  ASDP),  and  the  description  of  interface  considerations  (Section  5  of  the  ASDP). 

3.5  Analysis  of  Delivery  System  Capabilities 

The  analysis  of  delivery  system  capabilities  includes  the  hardware  and  software 
configuration  for  the  instructor  station,  the  student  stations,  the  classroom  furniture,  other 
automated  classroom  components,  and  the  local  area  network  components.  The  purpose  of  the 
analysis  is  to  evaluate  alternative  delivery  system  configurations  in  terms  of  costs  and  capabilities. 
This  analysis  is  directly  relevant  to  the  preparation  of  an  ASDP.  The  analysis  should  result  in  a 
statement  of  the  best  system  configuration  (Section  2  of  the  ASDP)  and  a  description  of  the 
alternative  configurations  that  were  considered  (Section  3  of  the  ASDP). 

The  current  lETM/AC  configuration  includes  a  LiveBoard  (an  electronic  blackboard)  for 
classroom  presentations  and  a  computer-controlled  LINK  video  networking  system  to  allow  the 
instructors  to  control  the  student  station  displays.  An  analysis  of  the  course  requirements  should  be 
conducted  to  determine  if  these  delivery  system  components  are  necessary  and,  if  so,  what  features 
they  should  provide. 

The  instructor  station  requirements  should  specify  the  CPU  configuration  including 
memory,  speed,  sound  card,  modem,  a  CD-ROM  player,  tape  backup,  operating  system,  COTS 
software,  customized  applications  software,  input  devices,  size  of  monitor,  miscellaneous 
peripherals  such  as  printers  and  surge  suppressors.  In  determining  these  specifications,  it  is 
essential  to  decide  whether  the  lETMs  will  be  stored  and  executed  from  the  CD-ROM  player  or 
from  the  hard  drive. 

The  network  requirements  should  specify  the  hardware  and  software  configuration 
(instructor  station  system  or  separate  file  server  system)  including  the  LAN  operating  system,  file 
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server  capacity,  number  and  location  of  users,  application  software,  processing  requirements, 
wiring/installation,  and  communication  interfaces. 

The  specification  for  the  student  stations  should  be  based  on  the  specifications  for  the 
instructor  station  and  network.  The  student  station  requirements  should  specify  the  CPU 
configuration  including  memory,  speed,  sound  cards,  a  CD-ROM  player,  operating  system,  input 
device  and  COTS  run  time  software. 

The  specifications  for  the  delivery  system  should  also  include  the  necessary  classroom 
furniture  to  support  the  configuration.  Furniture  includes  the  type  and  number  of  student  work 
stations,  the  type  of  instructor  station,  an  instructor  podium,  chairs,  stools,  tables,  printer  stands, 
and  portable  units  for  use  in  laboratories  or  other  classrooms.  The  costs  for  alternative  features 
should  be  analyzed  to  determine  the  best  value  to  meet  the  requirements. 

3.6  Analysis  of  Interface  Considerations 

When  a  course  is  converted  to  lETM/AC  technology,  it  may  have  an  impact  on  systems, 
procedures,  and  organizations  that  are  both  internal  and  external  to  the  schoolhouse.  For  the  most 
part,  the  impact  within  the  schoolhouse  is  discussed  above  in  the  analysis  of  course  management 
requirements.  Additional  interface  considerations  within  the  schoolhouse  may  include  such  things 
as  networking  several  classrooms  to  a  central  file  server  system  to  distribute  training. 

The  impact  of  the  lETM/AC  technology  on  the  interfaces  with  infrastructure  outside  the 
schoolhouse  should  also  be  considered.  The  infrastructure  will  vary  from  one  organization  to  the 
next  and  so  will  the  interface  issues.  To  facilitate  the  analysis,  the  concerns  that  need  to  be 
addressed  have  been  organized  into  general  categories.  A  representative  set  of  questions  is 
provided  below  for  each  category. 

3.6.1  Facilities 

•  What  classroom  space  is  available?  What  kind  of  site  preparation  and  refurbishment  is 
required  to  support  the  hardware/software  configuration? 

•  Are  there  command  or  base  contracts  available  to  obtain  contractor  support  for  site 
preparation,  classroom  refurbishment,  equipment  installation,  cabling,  etc.? 

•  What  provisions  are  there  for  classroom  facility  maintenance  (i.e.,  air  conditioning 
service,  filters,  cabling)? 

•  Are  there  requirements  for  integration  with  the  Learning  Resource  Center  (LRC)? 

3.6.2  Testing  and  Security 

•  What  kind  of  testing  procedures  need  to  be  established  to  verify  system  operations  and 
reliability?  Who  should  participate?  When  should  it  be  done? 

•  What  kind  of  physical  classroom  security  procedures  need  to  be  established? 
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•  What  kind  of  ADP  security  procedures  need  to  be  established? 

•  Is  classified  material  involved? 

3.6.3  Hardware/Equipment  Acquisition  and  Maintenance 

•  What  procedures  are  established  for  acquiring  both  customized  and  COTS  hardware? 

•  Can  the  small  multi-user  computer  contract  be  used  for  COTS  components?  What 
Contract  Line  Item  Numbers  (CLINs)  meet  the  requirements? 

•  Are  other  command  or  base  contracts  available  for  quantity  buys? 

•  Can  some  items  (i.e.,  computer  workstations  and  chairs)  be  purchased  through  a  GSA 
contract? 

•  Are  other  Navy  contracts  available  to  acquire  specific  hardware/equipment  used  in  the 
automated  classrooms?  If  so,  what  are  the  procedures  for  using  them?  How  do  require¬ 
ments  get  integrated? 

•  Does  the  computer  hardware  come  with  a  warranty?  If  so,  how  long? 

•  Does  the  vendor  offer  a  maintenance  agreement  (i.e.,  on-call  service,  field  support, 
repair  and  replace)?  Are  hourly  rates  charged? 

•  Is  there  a  command  or  base  ADP  group  responsible  for  preventive  and/or  corrective 
maintenance  for  microcomputers  and  peripherals? 

•  Should  the  acquisition  include  provisioning  for  spares  for  certain  components  to  mini¬ 
mize  downtime?  If  so,  what  components? 

3.6.4  Networking 

•  What  communications  components  are  required? 

•  Are  there  specific  LAN  related  interface  and  gateway  requirements? 

•  Is  a  wide  area  network  (WAN)  available? 

•  Is  connection  to  a  host  processor  at  a  central  site  available  or  desirable? 

3.6.5  Software  Acquisition  and  Upgrades 

•  What  procedures  are  established  for  acquiring  both  customized  and  COTS  software? 

•  Can  the  small  multi-user  computer  contract  be  used  for  COTS  software  packages? 
What  Contract  Line  Item  Numbers  (CLINs)  meet  the  requirements? 

•  Are  other  command  or  base  contracts  available  for  quantity  buys? 
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•  Are  software  site  licenses  available?  Have  they  already  been  acquired  at  a  command 
or  base  level?  Who  is  responsible  for  managing  software  licenses? 

•  What  procedures  are  established  for  software  upgrades? 

•  Are  other  Navy  contracts  available  to  develop  customized  software?  If  so,  what  are  the 
procedures  for  using  them?  How  do  requirements  get  integrated? 

•  Is  there  customized  software  that  needs  to  be  maintained  and  revised  as  requirements 
change? 

3.6.6  Personnel  Training/Logistic  Support 

•  What  procedures  are  established  for  acquiring  the  necessary  personnel  skill  training 
(i.e.,  ADP  administration,  LAN  administration,  applications  software)? 

•  Does  the  capability  to  provide  systems  administration  and  LAN  administration  sup¬ 
port  already  reside  on  base?  Can  the  school  requirements  be  integrated? 

•  Are  other  command  or  base  contracts  available  to  provide  contractor  support  for  sys¬ 
tems  administration  and  LAN  administration? 

•  Is  there  a  contractor  or  Navy  activity  responsible  for  maintaining  the  lETM  database? 
What  are  the  procedures  and  schedule  for  receiving  revisions  in  the  schoolhouse? 

•  Do  the  requirements  for  the  interface  with  the  Technical  Library  change? 

•  Do  the  requirements  for  the  interface  with  the  Learning  Resource  Center  (LRC) 
change? 

•  Is  there  a  contract  available  for  CD-ROM  mastery  and  production?  If  so,  what  are  the 
costs?  How  are  requirements  integrated? 

This  analysis  should  be  conducted  to  ensure  cost  effective  implementation  and  life  cycle 
support  of  the  lETM/AC  technology  in  the  schoolhouse.  The  results  of  the  analysis  of  interface 
considerations  are  directly  relevant  to  preparing  a  number  of  sections  in  an  ASDP.  The  interface 
options  and  requirements  impact  the  determination  of  the  best  approach  (Section  2),  the  description 
of  the  alternatives  that  were  considered  (Section  3),  the  development  of  cost  estimates  (Section  4), 
the  statement  of  the  system  interfaces  (Section  5),  the  plan  for  testing  (Section  6),  and  the  definition 
of  additional  requirements  (Section  8). 

3.7  Development  of  Non-Recurring  and  Recurring  Cost  Estimates 

The  guidelines  for  preparing  an  ASDP  call  for  the  non-recurring  (one  time)  and  the 
recurring  (annual)  cost  estimates  to  be  presented  in  a  summary  format.  The  costs  for  the  proposed 
solution  are  displayed  first  and  then  the  costs  for  alternative  solutions  (if  they  were  evaluated  on 
the  basis  of  cost)  are  displayed  using  the  same  cost  elements  to  facilitate  comparison.  The  scope 
and  complexity  of  the  project  determine  the  cost  elements  and  format  to  use.  Generally,  life  cycle 
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costs,  cost  avoidance  amounts,  and  return  on  investment  figures  are  expressed  in  current  fiscal  year 
dollars.  However,  a  present  value  analysis  may  be  used  to  display  the  costs  over  the  system  life 
cycle. 

Although  the  ASDP  guidelines  require  only  a  summary  display  of  major  cost  elements,  an 
underlying  life  cycle  cost  model  (LCCM)  that  documents  the  results  of  the  analyses  described  in 
Sections  3.2  through  3.6  should  be  used  to  derive  the  summary  costs  that  are  presented.  As  part  of 
this  study,  we  developed  a  generic  LCCM  for  analyzing  the  costs  for  implementing  lETM/AC 
technology  in  other  NAVEDTRACOM  schools.  The  generic  LCCM  was  designed  using  LOTUS 
1.2.3,  Release  4  for  Windows,  but  any  other  spreadsheet  software  could  be  used.  The  use  of 
spreadsheet  software  makes  it  relatively  easy  to  refine  the  structure  of  critical  cost  elements  and 
revise  the  cost  data  presented.  It  also  highlights  the  relationships  between  cost  elements  and  allows 
the  impact  of  changes  in  cost  estimates  (i.e.,  minimum-maximum  ranges)  to  be  evaluated. 

The  generic  LCCM  is  organized  into  two  parts:  non-recurring  and  recurring  costs.  For  the 
purpose  of  this  study,  non-recurring  costs  are  defined  as  the  one  time  costs  projected  for  initial 
acquisition  and  implementation  of  the  lETM/AC  technology  in  the  schoolhouse.  The  LCCM  for 
non-recurring  costs  is  shown  in  Table  3.  Recurring  costs  are  defined  as  the  annual  life  cycle 
management  costs  for  operating  and  maintaining  the  technology.  The  LCCM  for  recurring  costs  is 
shown  in  Table  4. 

The  cost  elements  used  in  this  study  are  broad  in  scope  to  meet  the  diverse  requirements  of 
future  implementations.  They  consist  of  a  number  of  sub-elements  to  help  clarify  the  results  of  the 
costs/benefits  analyses  described  previously.  The  following  subsections  define  the  non-recurring 
and  recurring  costs  elements  displayed  in  the  generic  LCCM  shown  in  Tables  3  and  4,  respectively. 
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Table  3 


Life  Cycle  Cost  Model:  Non-Recurring  Costs 


Configuration  1 

Configuration  2 

Non-Recurring 

Cost  Elements 

Unit  Cost 

Quantity 

Total 

Cost 

Unit  Cost 

Quantity 

Total 

Cost 

Student  Station 

Hardware/Software 

Work  Station  Furniture 

Instructor  Station 

Hardware/Software 

Work  Station  Furniture 

Local  Area  Network 

LiveBoard 

Link  System 

Basic  Configuration 

Optional  Features 

Classroom  Preparation 

Designs 

Computer  Room  Flooring 

Installation 

Hardware  Installation 

User  Trainii^ 

Course  Conversion 

Total  Non-Recurring  Costs 

Table  4 


Life  Cycle  Cost  Model:  Recurring  Costs 


Recurring 

Cost  Elements 

Configuration  1 

Configuration  2 

Unit  Cost 

Quantity 

Unit  Cost 

Quantity 

Total 

Cost 

Hardware  Maintenance 

Software  Maintenance 

Hardware/Software  Upgrades 

Logistical  Support 

System/Network  Administration 

User  IVaining 

Course  Revision 

Total  Recurring  Costs 
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3.7.1  Non-Recurring  Cost  Elements 

The  non-recurring  cost  elements  that  are  usually  used  to  derive  the  costs  for  implementing 
a  new  computer-based  training  system  also  apply  to  the  lETM/AC  technology.  Generally,  the  costs 
elements  cover  hardware  and  software  acquisition,  installation,  and  courseware  development 
costs.  For  the  lETM/AC  technology,  the  acquisition  costs  include  hardware,  software,  other 
automated  classroom  components,  local  area  network  (LAN)  components,  user  documentation, 
spares,  classroom  furniture,  and  user  training.  Installation  costs  include  classroom  design  and 
preparation,  refurbishment  of  the  facility  space  (if  necessary),  physical  installation  of  the  hardware 
and  network  components.  Courseware  development  costs  cover  course  conversion  to  the  lETM/ 
AC  format. 

To  make  the  LCCM  more  useful  in  determining  the  costs  for  future  automated  classroom 
implementations,  the  estimates  for  non-recurring  costs  are  aggregated  into  the  cost  elements 
defined  below. 

Student  Station.  The  configuration  of  a  student  station  may  vary  depending  upon  the 
characteristics  of  the  training  to  be  delivered.  The  number  of  student  stations  required  in  a  given 
automated  classroom  may  vary  depending  on  student  throughput,  course  convenes,  classroom 
availability  and  other  facility  constraints.  The  sub-elements  include: 

•  Computer  Hardware  Configuration. 

•  Software  Configuration  (including  COTS  and  customized  applications). 

•  Student  Workstation  and  Chair. 

•  User  Documentation  (if  separately  priced). 

•  Spares  Provisioning  (as  applicable). 

•  Installation  (as  applicable). 

Instructor  Station.  The  configuration  of  an  instructor  station  may  vary  depending  upon  the 
characteristics  of  the  training  to  be  delivered  and  classroom  facility  constraints.  The  sub-elements 
include: 

•  Computer  Hardware  Configuration. 

•  Laser  Printer. 

•  Software  Configuration  (including  COTS  and  customized  applications). 

•  Instructor  Workstation  and  Chair  or  Stool. 

•  Instructor  Podium. 

•  User  Documentation  (if  separately  priced). 
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•  Spares  Provisioning  (as  applicable). 

•  Installation  (as  applicable). 

Automated  Classroom  Components.  The  components  to  be  included  in  an  automated 
classroom  may  vary  depending  upon  the  requirements  for  instructional  delivery  and  management 
and  any  constraints  on  classroom  facilities.  The  sub-elements  include: 

•  LiveBoard  (Electronic  blackboard). 

•  LINK  System  (Computer-controlled  video  networking). 

•  Local  Area  Network  (LAN). 

•  Other  Components  or  Equipment. 

•  User  Documentation  (if  separately  priced). 

•  User  Training  (as  available). 

•  Spares  Provisioning  (as  applicable). 

•  Installation  (as  applicable). 

Classroom  Preparation.  The  costs  to  prepare  a  classroom  are  a  function  of  the  designated 
space  at  a  given  facility  and  the  delivery  system  configuration  to  be  installed.  For  example, 
minimal  costs  would  be  incurred  in  upgrading  a  computer-based  training  classroom  to  support  the 
lETM/AC  technology.  Major  costs  would  be  incurred  when  a  space  requires  complete 
refurbishment  Classroom  preparation  may  include  items  such  as  facility  design,  demolition, 
electronic  power  service  upgrades,  electrical  power  conditioning  (GFI),  power  backup  units, 
lighting  fixture  upgrades,  environmental  and  climate  control  systems  (controls,  cooling/heating 
equipment,  vent  ducting,  weather  seals,  registers/diffusers,  dust/dirt  air  scrubbers,  anti-static 
conditioners),  paint,  cable  wiring  and  connectors,  and  computer  room  flooring. 

Hardware  Installation.  The  costs  to  physically  install  the  hardware,  furniture,  LAN,  and 
other  automated  classroom  components  may  be  separate  or  they  may  be  included  in  the  cost 
elements  shown  above. 

User  Ikaining.  The  requirements  and  costs  for  user  training  are  a  function  of  the  automated 
classroom  configuration.  The  sub-elements  include: 

•  Instructor  Training. 

•  Systems  Administrator  Training. 

•  LAN  Administrator  Training. 

•  Software  Applications  Training. 
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Course  Conversion.  The  costs  to  convert  the  course  are  a  function  of  the  course 
characteristics,  the  amount  of  redesign,  the  authoring  system  capabilities,  and  the  personnel 
resources  available.  The  sub-elements  include: 

•  IG  and  TG  Revision. 

•  Instructional  Media  Development. 

3.7.2  Recurring  Cost  Elements 

Generally,  the  recurring  cost  elements  used  to  derive  annual  life  cycle  support  cost 
estimates  for  a  computer-based  training  system  apply  to  the  lETM/AC  technology.  Relevant  costs 
include  hardware  maintenance,  hardware  upgrades,  customized  software  maintenance  and 
enhancements,  software  upgrades,  logistical  support,  system  administration,  network 
administration,  user  training,  and  courseware  revision  costs.  To  make  the  LCCM  more  useful  in 
determining  life  cycle  costs  for  future  automated  classroom  implementations,  the  recurring  costs 
are  aggregated  into  the  cost  elements  defined  below. 

Hardware  Maintenance.  This  element  refers  to  the  costs  for  maintaining  the  hardware  for 
the  instructor  station,  student  station,  and  other  automated  classroom  components.  One  or  several 
of  the  following  sub-elements  may  apply: 

•  Hardware  Warranty  (part  of  the  acquisition  costs). 

•  On-Site  or  On-Call  Field  Service  (by  the  vendor). 

•  Spares  Provisioning  (swap  out  components  and  return  defects  to  vendor). 

•  Command  ADP  Service  Center  (in-house  group  responsible  for  corrective  mainte¬ 
nance). 

Software  Maintenance  and  Enhancements.  This  element  refers  to  the  costs  for 
maintaining  and  enhancing  any  customized  software  included  in  the  classroom  configuration. 

Hardware/Software  Upgrades.  This  element  refers  to  the  costs  for  upgrading  COTS 
hardware  components  (i.e.,  Pentium  processors,  increased  memory)  and  COTS  software  packages. 
As  technology  advances,  both  the  hardware  and  software  capabilities  will  increase  and  upgrades 
are  an  inexpensive  way  to  keep  pace. 

Logistical  Support  This  element  refers  to  the  costs  for  maintaining  training  consumables 
and  supplies;  storing  and  reproducing  training  materials;  updating,  maintaining,  reproducing,  and 
distributing  the  technical  manuals;  pressing  and  replicating  CD-ROMs;  and  special  classroom 
requirements  (i.e.,  air  conditioning  filters,  light  bulbs). 

System/Network  Administration.  This  element  refers  to  the  personnel  costs  for 
administering  the  lETM/AC  classrooms  and  LANs.  The  mix  and  number  of  personnel  are  a 
function  of  the  requirements  at  the  schoolhouse. 
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User  lyaining.  This  element  refers  to  the  on-going  requirement  for  training  as  personnel 
change  and  the  classroom  configuration  is  upgraded.  The  sub-elements  include: 

•  Instructor  Training. 

•  Systems  Administrator  Training. 

•  LAN  Administrator  Training. 

•  Software  Applications  Training. 

Course  Revision.  This  element  refers  to  the  costs  for  revising  the  instruction  (IGs,  TGs, 
and  other  media)  due  to  content  deficiencies  or  changes  in  technical  content,  including  updates  to 
the  lETM  database. 

3.8  Identification  of  Benefits 

The  final  activity  in  the  process  of  analyzing  the  costs  and  benefits  of  implementing  the 
lETM/AC  technology  in  NAVEDTRACOM  schools  is  to  identify  both  the  quantifiable  and  the 
nonquantifiable  benefits.  The  guidelines  for  preparing  Section  4  of  an  ASDP  suggest  five  benefit 
areas  that  may  be  relevant  in  justifying  the  proposed  solution.  These  general  benefit  areas  are 
defined  as  follows: 

1.  Cost  avoidance— an  increase  in  the  operational  efficiency  of  personnel  by  reducing  the 
time  spent  performing  specific  functions. 

2.  Cost  savings  or  cost  offsets— direct  reduction  in  operational  costs  and  identification  of 
specific  areas  that  will  be  reduced  or  eliminated  as  a  result  of  implementing  the  proposed  solution. 

3.  Increased  productivity— description  of  improvement,  quantified  in  labor  hours  or 
workload. 

4.  Quality  and  accuracy— description  of  improvement,  such  as  a  reduction  in  error  rate. 

5.  Manageability— description  of  improvement,  such  as  a  savings  in  the  number  of  labor 

hours. 


To  augment  these  general  benefit  areas  and  assist  in  developing  the  statement  of  benefits, 
we  offer  the  following  list  of  benefits  that  may  be  realized  by  implementing  the  lETM/AC 
technology: 

•  Decrease  in  student  training  costs  by  reducing  course  length. 

•  Lower  cost  per  student  per  course  by  increasing  student  throughput. 

•  Decrease  in  instructor  class  preparation  time. 

•  Improvements  in  student  performance  scores. 
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•  Reduction  in  student  academic  attrition  and  setbacks. 

•  Reduction  in  media  development  and/or  revision  time. 

•  Reduction  in  course  management  and  distribution  costs. 

Decrease  in  costs  for  course  supplies/instructional  media  (i.e.,  avoid  costs  for  paper 
reproduction,  inventory,  and  storage). 


Section  Four 


Costs/Benefits  Model 


4.0  Cost  Benefits  Model 


4.1  Scope 

As  documented  in  the  Evaluation  and  Implementation  Plan  (EIP)  for  the  joint  project, 
NAVSEA  was  responsible  for  providing  the  funding  and  support  to  implement  the  lETM/AC 
technology  in  the  GSE/GSM  classrooms  for  field  testing  and  evaluation.  NPRDC  was  tasked  to 
evaluate  Ae  effectiveness  of  potential  enhancements  to  the  technology  and  determine  the 
efficiencies  of  the  automated  classroom.  Based  on  the  results  of  the  evaluation,  NPRDC  and 
NAVSEA  are  responsible  for  developing  specifications  and  recommendations  for  future 
implementations.  CNET  and  NAVSEA  are  responsible  for  providing  support  to  transition  the 
products  into  the  training  environment.  The  costs/benefits  model  presented  in  this  section  was 
developed  as  part  of  the  evaluation. 

The  costs/benefits  model  for  the  GS  “C”  School  supports  the  format  requirements  for 
Section  4  of  the  ASDP  (Costs  and  Benefits).  The  model  has  three  components:  (1)  the  statement  of 
benefits,  including  both  qualitative  and  quantitative  benefits,  (2)  the  LCCM  for  non-recurring  costs 
described  in  Section  3.7.1,  and  (3)  the  LCCM  for  annual  recurring  costs  described  in  Section  3.7.2. 

As  part  of  the  analysis,  we  assigned  confidence  levels  to  the  cost  estimates  developed  for 
the  costs/benefits  model.  The  alphabetical  designator  and  definition  for  the  different  levels  of 
confidence  (taken  from  OPNAV  Instruction  7000. 17A)  are  listed  below. 

A.  Detailed  Cost  Estimate  based  on  contract  plans  and  evaluation  of  firm  quotes  for  major 
material  items. 

B.  Bid  Evaluation  Cost  Estimate  based  on  contract  plans  and  evaluation  of  contractor 
proposals  in  response  to  an  RFP. 

C.  Budget  Quality  Estimate  based  on  engineering  analysis  of  detailed  characteristics  of  item 
under  consideration. 

D.  Feasibility  Estimate  based  on  technical  feasibility  studies  and/or  extrapolated  from  higher 
quality  cost  estimates  of  similar  items. 

E.  Computer  Estimate  based  on  cost  estimating  relationships  and  gross  parameters. 

F.  Ball  Park  Estimates  prepared  in  absence  of  minimum  design  and  cost  information  and 
based  on  gross  parameters. 

The  cost  estimates  developed  for  the  quantifiable  benefits  were  assigned  a  confidence  level 
of  “C”  (Budget  Quality  Estimate)  since  they  are  based  on  student  throughput  projections  and  other 
assumptions.  The  cost  estimates  developed  for  the  LCCM  for  non-recurring  costs  were  assigned  a 
confidence  level  of  “A”  (Detailed  Cost  Estimate)  since  they  are  based  on  costs  provided  by  the 
Hardware  Installation  Activity  responsible  for  acquiring  the  classroom  hardware  and  software 
configuration. 
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We  did  not  develop  any  cost  estimates  for  the  LCCM  for  recurring  costs.  As  described  in 
Section  5.1.1,  several  Navy  organizations  are  in  the  process  of  specifying  the  life  cycle 
management  requirements  for  the  automated  classrooms.  However,  specific  statements  about  the 
life  cycle  support  requirements  were  not  available  and,  in  the  absence  of  detailed  information,  we 
could  not  derive  any  meaningful  annual  recurring  cost  estimates.  In  light  of  the  on-going  efforts, 
the  development  of  ball  park  estimates  (a  confidence  level  of  “F’)  did  not  seemed  warranted. 

Section  4.2  presents  the  statement  of  benefits.  The  statement  of  benefits  provides  a 
comparison  between  converting  the  conventional  paper-based  courses  to  the  lETM/AC  technology 
and  maintaining  the  status  quo.  Section  4.3  presents  the  non-recurring  cost  estimates  to  acquire  and 
install  the  hardware  and  software  configuration  implemented  in  the  five  GSE/GSM  automated 
classrooms.  The  non-recurring  cost  estimates  are  displayed  in  the  LCCM  format  previously  shown 
in  Table  3.  Section  4.4  provides  general  statements  about  the  life  cycle  support  requirements  for 
the  GSE/GSM  automated  classrooms.  There  are  no  recurring  cost  estimates  to  display  in  the 
LCCM  format  previously  shown  in  Table  4. 

4.2  Statement  of  Benefits 

Both  Section  3.8  and  Section  5.2  discuss  potential  benefits  that  may  be  realized  by 
implementing  the  lETM/AC  technology  in  the  schoolhouse.  This  section  presents  the  immediate 
benefits  that  could  be  analyzed  and  quantified  in  a  meaningful  way  of  implementing  the  lETM/AC 
technology  in  the  GSE/GSM  courses.  These  benefits  are  summarized  below: 

1.  Improvements  in  student  performance  scores.  Initial  data  indicates  a  slight 
improvement  in  student  performance  scores  in  the  lETM-based  courses  over  student  performance 
scores  in  the  paper-based  courses.  These  preliminary  results  suggest  that  the  introduction  of  the 
lETM/AC  technology  and  subsequent  reduction  in  course  length  has  not  had  an  adverse  effect  on 
student  performance. 

2.  Reduction  in  student  training  costs.  The  elimination  of  seven  training  weeks  from 
the  GSE  pipeline  and  three  training  weeks  from  the  GSM  pipeline  results  in  a  reduction  in  student 
training  costs  of  over  $1,800,000  for  FY  95  and  FY  96.  The  assumptions  used  to  develop  these 
cost  estimates  were  conservative  to  avoid  overstating  the  potential  benefits.  The  actual  savings 
may  be  much  larger  than  those  stated  here. 

3.  Reduction  in  course  reproduction  costs.  The  cost  savings  for  not  reproducing  the 
paper-based  technical  manuals  used  in  the  GSE/GSM  courses  is  estimated  at  over  $96,000  for  FY 
96.  Again,  the  assumptions  used  to  develop  this  were  conservative  to  avoid  overstating  the  poten¬ 
tial  benefits.  The  actual  savings  may  be  much  larger  than  those  stated  here. 

In  addition  to  these  quantifiable  benefits,  there  are  other  important  benefits  that  impact  the 
quality  of  the  training.  From  the  perspective  of  SSC  GLARES  personnel,  one  of  the  most 
significant  benefits  of  the  lETM  technology  is  the  capability  to  have  current,  up-to-date  technical 
manuals  available  for  operational  use  in  the  fleet  and  for  training  in  the  classroom.  The  process  of 
preparing  and  disseminating  paper  changes  to  users  and  having  users  incorporate  the  changes  and 
maintain  the  paper-based  manuals  is  time  consuming,  labor  intensive  and  costly.  Incorporating  the 
lETM  technology  into  the  automated  classrooms  allows  technicians  attending  the  formal  school  to 
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receive  technical  training  using  the  same  accurate  and  current  documentation  available  in  the  fleet. 
It  also  provides  them  with  training  on  how  to  use  the  lETM  technology  when  they  are  back  on  the 
job. 


The  benefits  gained  as  a  result  of  the  GSE/GSM  implementation  relate  directly  to  the  initial 
decision  to  implement  the  lETM/AC  technology  in  the  GS  “C”  School.  As  discussed  in  Section 
1.3.1,  NAVSEA  selected  the  gas  turbine  systems  for  development  of  an  integrated  lETM  database 
for  use  aboard  ship  and  in  the  schoolhouse  for  a  variety  of  reasons.  The  logistic  problems  inherent 
in  maintaining,  updating,  and  distributing  massive  volumes  of  paper-based  technical  manuals 
made  the  gas  turbine  systems  a  prime  candidate  for  automation.  One  consideration  was  to  use 
advances  in  technology  to  improve  the  process  of  managing  technical  manuals  and  keeping 
technical  information  acctirate  and  current  for  operational  use  in  the  fleet  and  in  training. 

A  major  consideration  for  selecting  the  GS  “C”  School  as  a  field  test  site  was  the  extensive 
use  of  large  sets  of  technical  manuals  in  the  advanced  GSE  and  GSM  courses.  The  benefits  to  be 
gained  by  replacing  paper  materials  with  on-line  electronically  formatted  materials  appeared 
significant  in  terms  of  botti  the  costs  and  personnel  time  required  for  to  manage,  store,  distribute, 
and  update  the  materials  used  in  the  classroom.  Another  consideration  for  introducing  lETMs  was 
the  potential  of  reducing  student  training  costs  by  reducing  the  length  of  the  GSE  and  GSM 
courses.  However,  the  question  was  whether  the  current  levels  of  student  proficiency  could  be 
maintained  or  improved  with  the  reductions  in  course  length. 

Section  4.2.1  discusses  the  impact  of  the  technology  on  student  performance  scores. 
Section  4.2.2  provides  a  detailed  description  of  the  reduction  in  student  training  costs  estimated  for 
the  current  fiscal  year  and  projected  for  FY96.  Section  4.2.3  presents  the  assumptions  used  in 
deriving  the  estimate  of  the  savings  in  course  reproduction  costs  for  FY96. 

4.2.1  Benefit:  Improvements  in  Student  Performance  Scores 

The  comparison  of  student  performance  scores  in  the  traditional  paper-based  courses  and 
the  lETM-based  courses  is  still  in  process,  but  the  preliminary  results  of  the  analysis  look 
promising.  The  test  scores  for  six  paper-based  GSE  and  GSM  classes  were  obtained  to  use  as  the 
baseline.  Test  scores  for  seven  GSE  lETM-based  classes  and  three  GSM  lETM-based  classes  were 
available  for  comparison  at  the  time  this  report  was  prepared.  The  preliminary  comparisons  of  GSE 
and  GSM  student  performance  scores  are  summarized  in  Tables  5  and  6,  respectively. 

As  shown  in  Table  5,  even  though  the  number  of  students  is  small,  there  is  a  slight 
improvement  (approximately  three  points  out  of  100)  in  the  overall  average  student  performance 
scores  for  the  GSE  lETM/ AC  classes.  As  shown  in  Table  6,  with  only  three  lETM/ AC  class 
graduations  to  date,  there  is  also  a  slight  improvement  (approximately  two  points  out  of  100)  in  the 
overall  average  student  performance  scores  for  the  GSM  lETM/AC  classes.  These  preliminary 
results  suggest  the  introduction  of  the  lETM/AC  technology  and  subsequent  reduction  in  course 
length  has  not  had  an  adverse  effect  on  student  performance.  The  data  for  academic  attrites  and 
student  setbacks  was  not  sufficient  to  make  any  comparisons  between  the  paper-based  and  lETM- 
based  courses. 
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Table  5 


Comparison  of  GSE  Student  Performance  Scores 


Paper-based  Classes 

Average  Class  Score 

lETM/AC  Classes 

Average  Class  Score 

93090  (A^=  13) 

87.32 

94120  (A  =  3) 

94.77 

93100  (Y=  11) 

92.55 

941300V  =10) 

93.32 

94010  (Y=  13) 

90.23 

94135  (iV=  7) 

91.09 

94020  (Y=  12) 

89.43 

95010  0V=  5) 

85.60 

94030  (Y=  15) 

84.49 

95030  (1V=  3) 

95.30 

94085  (Y=  6) 

90.03 

95040  (iV=  3) 

94.30 

95045  (IV  =  2) 

89.50 

95050  (1V=  5) 

In  process 

Number  of  Students 

Overall  Average  Score 

Number  of  Students 

Overall  Average  Score 

70 

88.66 

38 

91.88 

Table  6 

Comparison  of  GSM  Student  Performance  Scores 

P^r-based  Classes 

Average  Class  Score 

94065  (A  =14) 

95030  (A  =  7) 

93.90 

94075  (A^=  5) 

93.72 

95040  (A  =  9) 

88.60 

94100  (iV=  8) 

89.74 

95045  (A  =12) 

95.80 

94110  (iV=  9) 

89.82 

95050  (A  =15) 

In  process 

94120  (A  =  7) 

91.56 

95060  (1V=  6) 

In  process 

94145  (Y=  9) 

92.54 

Number  of  Students 

Overall  Average  Score 

Number  of  Students 

Overall  Average  Score 

52 

90.60 

49 

92.23 

4.2.2  Benefit:  Reduction  in  Student  Training  Costs 

The  potential  savings  in  student  training  costs  are  a  function  of  the  average  cost  per  student 
per  week  per  course,  the  projected  number  of  students  to  go  through  each  course,  and  the  reduction 
in  training  weeks  between  the  paper-based  courses  and  the  lETM -based  courses.  The  estimates  of 
savings  in  student  training  costs  presented  in  Table  7  were  assigned  a  confidence  level  of  “C” 
(Budget  Quality  Estimate)  since  they  are  based  on  student  throughput  projections  for  FY95  and 
FY96.  The  potential  student  training  cost  savings  will  vary  as  the  student  throughput  figures 
change.  If  the  actual  throughput  is  less  than  the  projected  throughput  figures,  the  cost  savings  will 
be  less. 


The  FY95  student  throughput  figures  were  based  on  actual  throughput  to  date  and  projected 
throughput  for  the  remainder  of  the  fiscal  year.  The  FY95  throughput  for  the  two  GSE  courses  is 
38  students.  This  figure  is  the  same  as  the  number  of  students  shown  in  the  GSE  lETM-based 
courses  in  Table  5.  The  FY95  throughput  for  the  GSM  course  is  49  students.  This  figure  is  the  same 
as  the  number  of  students  shown  in  the  GSM  lETM-based  course  in  Table  6.  The  FY96  throughput 
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figures  were  based  on  the  latest  projections  provided  to  SSC  GLAKES  school  management 
personnel. 


Table? 

Student  IVaining  Cost  Savings  for  GSEVGSM  Courses 


Throughput 

FY95  FY96 

Weekly  Cost  per 

Student  Number  of 

($)  Weeks  per  Course 

FY  95  Student 
Training  Costs 
($) 

FY  96  Student 
Training  Costs 

(S) 

GSE  302  Course 

58 

72 

1,146.00 

Paper  5  weeks 

332,340.00 

412,560.00 

58 

72 

1,146.00 

IEIm  0  weeks 

0.00 

0.00 

Student  IVaining  Cost  Savings 

5  weeks 

332,340.00 

412,560.00 

GSE  328  Course 

58 

72 

1,199.80 

Paper  15  weeks 

683,886.00 

1,295,784.00 

58 

72 

1,199.80 

ETM  13  weeks 

592,701.20 

1,123,012.00 

Student  Training  Cost  Savings 

2  weeks 

91,184.80 

172,771,20 

GSM  303  Course 

49 

Paper  17  weeks 

1,132,880.00 

4,161,600.00 

49 

ETM  14  weeks 

932,960.00 

3,427,200.00 

Student  Training  Cost  Savings 

3  weeks 

199,920.00 

734,400.00 

Total  Student  Cost  Savings 

10  weeks 

623,444.80 

1,319,731.20 

The  average  cost  per  student  per  week  per  course  was  provided  by  personnel  in  the  New 
Technologies  Division  at  SSC  GLAKES.  These  estimates  are  displayed  in  current  fiscal  year 
dollars.  The  weekly  costs  include  student  salaries,  but  they  do  not  include  base  manning  or  base 
operational  costs.  The  weekly  costs  also  do  not  include  per  diem  costs  since  not  all  students 
attending  the  course  are  on  temporary  duty  assignment.  The  number  of  students  on  per  diem  will 
vary  with  each  course  convene.  If  student  per  diem  costs  were  included,  the  savings  would  be  even 
greater. 


As  part  of  the  effort  to  convert  the  curriculum  to  the  lETM/AC  format,  the  course 
supervisors  and  instructors  also  revised  and  shortened  the  GSE  and  GSM  training  pipelines.  The 
introduction  of  the  lETM/AC  technology  provided  an  opportunity  to  redesign  and  rewrite  some  of 
the  maintenance  courses  in  the  training  pipelines. 

Originally,  the  GSE  pipeline  consisted  of  one  core  course  (A-652-0302)  plus  three 
strands—the  FFG  maintenance  course  (A-652-0327),  the  CG/DD  maintenance  course  (A-652- 
0328),  and  AOE/DDG  maintenance  course  (A-652-0329).  Only  the  core  course  (GSE  302)  and  the 
CG/DD  maintenance  course  (GSE  328)  were  included  in  the  costs/benefits  analysis.  At  this  point, 
there  are  no  plans  to  convert  the  paper-based  FFG  course  (GSE  327)  to  the  lETM/AC  format.  The 
paper-based  AOE/DDG  course  (GSE  329)  is  in  the  process  of  being  converted.  The  GSM  pipeline 
consisted  of  one  core  course  (GSM  303)  which  was  included  in  the  costs/benefits  analysis. 
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As  shown  in  Table  7,  the  GSE  pipeline  was  reduced  by  seven  weeks  across  two  courses  and 
the  GSM  pipeline  was  reduced  by  three  weeks  in  one  course.  The  GSE  302  course  was  deleted 
from  the  GSE  pipeline.  Therefore,  this  five  week  paper-based  course  was  reduced  to  zero  weeks  in 
the  lETM  format.  The  15  week  paper-based  GSE  328  course  was  rewritten  to  include  two  weeks 
of  GSE  302  course  content.  However,  the  overall  length  of  the  lETM-based  GSE  328  course  was 
reduced  to  13  weeks.  Likewise,  the  17  week  paper-based  GSM  303  course  was  rewritten  to  add  two 
weeks  of  course  content.  But,  the  overall  length  of  the  lETM-based  GSM  303  course  was  reduced 
to  14  weeks. 

The  reduction  in  training  time  in  the  GSE  pipeline  was  attributed  to  two  factors;  (1)  the 
decrease  in  the  time  instructors  and  students  spend  using  the  paper-based  technical  manuals  in  the 
classroom  and  (2)  elimination  of  the  core  course  because  it  covered  materials  that  should  be  taught 
in  the  console  operator  courses  and  other  maintenance  courses.  The  reduction  in  training  time  in 
the  GSM  course  was  also  attributed  to  two  factors:  (1)  the  decrease  in  the  time  instructors  and 
students  spend  using  the  paper-based  technical  manuals  in  the  classroom  and  (2)  the  introduction 
of  computer-based  animations  to  visually  present  concepts  and  enhance  the  instruction. 

As  shown  in  Table  7,  the  total  savings  in  student  training  costs  for  the  three  courses  is 
$1,943,176.  The  elimination  of  the  GSE  302  course  results  in  $630,300  in  student  training  cost 
savings  for  FY95  and  FY96.  The  reduction  of  two  weeks  in  the  GSE  328  course  results  in  $263,956 
in  student  training  cost  savings  for  FY95  and  FY96.  The  reduction  of  three  weeks  in  the  GSM  303 
course  results  in  $934,320  in  student  training  cost  savings. 

Additional  savings  in  student  training  costs  may  be  realized  as  other  courses  are  redesigned 
for  the  lETM/AC  technology.  The  GSE  course  329  is  currently  being  rewritten.  A  Diesel  Division 
course  (A-652-0340)  with  an  lETM  is  currently  under  evaluation.  As  these  courses  are 
implemented  in  the  five  automated  classrooms,  the  potential  for  savings  in  student  training  costs 
increases. 

4.2.3  Benefit:  Reduction  in  Course  Reproduction  Costs 

A  major  aspect  of  managing  the  paper-based  GSE  and  GSM  course  materials  is 
maintaining  the  massive  sets  of  technical  manuals  used  in  both  the  classroom  and  laboratory 
instruction.  Since  only  two  master  sets  of  technical  manuals  are  provided  to  SSC  GLARES,  the 
technical  manuals  used  in  the  classroom  need  to  be  reproduced  at  the  local  Defense  Printing 
Service. 

For  the  purpose  of  this  analysis,  only  the  reproduction  costs  for  the  technical  manuals  are 
included.  Reproduction  costs  for  other  instructional  materials  (i.e.,  IG,  TGs,  training  aids)  are  not 
included.  The  costs  for  notebooks  and/or  binding  are  not  included.  The  costs  for  change  pages  and 
updates  that  may  be  issued  are  not  included.  The  costs  for  mailing  and  distribution  are  not  included. 
The  personnel  time  involved  in  putting  multiple  copies  of  the  technical  manuals  into  notebooks, 
keeping  them  updated,  and  storing  them  are  also  not  included. 

The  analysis  of  costs  for  reproducing  technical  manuals  for  the  paper-based  GSE  328 
course  and  the  paper-based  GSM  303  course  is  based  on  the  following  assumptions. 
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•  An  estimate  of  52  sets  of  technical  manuals  need  to  be  on-hand  for  each  course.  This 
assumes  that  the  typical  paper-based  classroom  is  set-up  to  accommodate  one  instruc¬ 
tor  and  up  to  25  students.  The  instructor  and  each  student  must  have  a  set  of  technical 
manuals.  At  least  two  classrooms  must  be  available  for  each  course  to  handle  overlap¬ 
ping  course  convenes 

•  The  reproduction  cost  is  an  average  of  $0.04  per  page.  The  cost  for  reproducing  stan¬ 
dard  8-12  X  11  pages  may  be  less,  but  the  costs  for  reproducing  fold-out  pages  may  be 
as  high  as  $0.18  per  foot. 

•  One  set  of  technical  manuals  for  GSE  328  course  includes  19,553  pages  contained  in 
34  volumes.  This  data  was  provided  by  the  Gas  Turbine  In-Service  Engineering  Activ¬ 
ity  (GS  ISEA)  responsible  for  maintaining  the  gas  turbine  technical  manuals  and  dis¬ 
seminating  the  manuals  and  updates  to  the  fleet  and  the  training  commands. 

•  One  set  of  technical  manuals  for  GSM  303  course  includes  26,608  pages  contained  in 
43  volumes.  This  data  was  provided  by  the  GS  ISEA  responsible  for  maintaining  the 
gas  turbine  technical  manuals  and  disseminating  the  manuals  and  updates  to  the  fleet 
and  the  training  commands. 

As  shown  in  Table  8,  the  estimate  of  the  total  savings  in  reproduction  costs  by 
implementing  the  lETM/AC  technology  in  the  GSE  and  GSM  courses  is  $96,015.  This  cost 
estimate  was  assigned  a  confidence  level  of  C  (Budget  Quality  Estimate).  Since  the  actuM  cost  per 
page  for  reproducing  the  manuals  may  vary  and  the  number  of  sets  of  manuals  required  is 
dependent  upon  course  scheduling  and  student  throughput,  it  is  difficult  to  assign  a  higher  level  of 
confidence.  However,  we  believe  this  is  a  conservative  estimate  of  the  reproduction  cost  savings 
that  may  be  realized  in  FY96. 


Table  8 

Reproduction  Cost  Savings  for  GSE/GSM  Courses 


Number  of  Sets 

Number  of 

Reproduction 

Total  Cost  for 

Course 

of 

Pages 

Cost  per  Page 

Reproduction 

Manuals 

per  Set 

($) 

($) 

GSE  328 

26 

19,553 

.04 

40,670 

GSM  303 

26 

26,608 

.04 

55,345 

Totals 

52 

46,161 

.04 

96, 015 

4.3  Life  Cycle  Cost  Model:  Non-Recurring  Costs 

NAVSEA  provided  the  support  for  funding  and  installing  the  lETM/AC  technology  at  SSC 
GLARES.  Five  automated  classrooms  were  established  for  the  GSE  and  GSM  courses  to  share. 
Five  instructor  stations  and  65  student  stations  were  acquired.  There  is  one  student  station  in  the 
laboratory  and  four  classrooms  with  12  student  stations  each  in  Building  238.  There  is  one  student 
station  in  the  laboratory  and  one  classroom  with  15  student  stations  in  Building  324. 
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SEA-04TD  was  responsible  for  refurbishing  the  classroom  spaces  and  acquiring  the 
automated  classroom  configuration  (including  computer  hardware,  basic  software,  and  furniture). 
The  Naval  Sea  Systems  Command  Automated  Data  Systems  Activity  (SEAADSA),  designated  as 
the  Hardware  Installation  Activity,  established  the  hardware  and  basic  software  configuration  for 
the  classrooms.  The  Naval  Sea  Logistics  Center,  Code  30  (SEALOG),  prepared  the  Equipment 
Facilities  Requirement  Plan  documenting  the  hardware/software  configuration  and  establishing  the 
facility  requirements.  The  acquisition  and  installation  activities  for  the  classrooms  were 
coordinated  by  SEAADSA  and  SEALOG. 

The  non-recurring  cost  estimates  for  implementing  the  lETM/AC  technology  in  the  GS  “C” 
School  are  presented  in  Table  9.  These  estimates  are  displayed  in  current  fiscal  year  dollars.  There 
were  no  alternative  configuration  to  cost  at  this  time.  However,  as  specifications  for  an  automated 
classroom  for  future  implementations  are  developed,  it  may  be  useful  to  compare  the  costs  of  the 
various  configurations. 

The  cost  for  acquiring  and  installing  the  five  GSE/GSM  automated  classrooms  was 
approximately  $2,227,000.  As  indicated  by  footnote  g  in  the  table,  this  figure  does  not  include  any 
personnel  costs  for  course  conversion.  These  personnel  costs  are  considered  to  be  implicit  costs. 
Swope  (1976)  defines  implicit  costs  as  those  costs  attributed  to  the  use  of  resources  that  are  already 
owned  for  which  there  are  no  explicit  budget  allocations  or  monetary  payments  required. 

The  cost  estimates  for  acquiring  the  instructor  station,  student  stations,  automated 
classroom  components,  and  preparing  and  refurbishing  the  classrooms  were  assigned  a  confidence 
level  of  A  (Detailed  Cost  Estimate).  These  costs  were  provided  by  personnel  from  the  New 
Technologies  Division  at  SSC  GLARES,  who,  in  turn,  received  the  cost  data  from  the  Hardware 
Installation  Activity  responsible  for  acquiring  and  maintaining  the  classroom  hardware  and 
software  configuration. 

Since  a  variety  of  issues  drove  the  analysis,  planning,  and  implementation  of  the  lETM/AC 
technology  in  the  GS  “C”  School,  the  cost  estimates  shown  in  Table  9  should  not  be  considered 
representative  of  the  costs  for  implementing  the  technology  in  other  NAVEDTRACOM  schools. 
For  example,  footnotes  c  and  d  in  the  table  identify  aspects  of  the  current  classroom  configuration 
where  there  could  be  significant  reductions  in  the  unit  costs.  Likewise,  footnote  b  illustrates  the 
decrease  in  costs  that  can  be  expected  as  advances  in  technology  are  made.  A  systematic  analysis 
of  costs  and  benefits  (like  the  process  described  in  Section  3)  should  yield  budget  quality  estimates 
of  the  potential  costs  for  implementing  the  technology  in  a  specific  schoolhouse  application. 
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Table  9 


GSE/GSM  Life  Cycle  Cost  Model:  Non-Recurring  Costs 


GSE/GSM  Configuration 

Non-Recurring 

Cost  Elements 

Unit  Cost 
($) 

Quantity 

Total  Cost 
($) 

Student  Station 

Hardware/Software 

8,277.00 

65 

538,005.00 

Work  Station  Furniture® 

5,867.00 

63 

359,621.00 

Instructor  Station 

Hardware/Software 

11,075.00 

5 

55,375.00 

Work  Station  Furniture 

7,533.00 

5 

37,665.00 

Local  Area  Network 

1,763.00 

5 

8,815.00 

LiveBoard*’ 

54,500.00 

5 

272,500.00 

Link  System^ 

73,995.00 

5 

369,975.00 

Classroom  Preparation** 

Designs 

13,000.00 

5 

65,000.00 

Computer  Room  Flooring 

35,000.00 

5 

175,000.00 

Installation 

67,000.00 

5 

335,000.00 

Hardware  Installation^ 

0;00 

0.00 

User  IVaining*^ 

0.00 

0.00 

Course  Conversion® 

Implicit  Cost 

Implicit  Cost 

Total  Non-Recurring  Costs 

2,226,956.00 

®Two  student  stations  are  portable  units  used  in  the  laboratories. 

^At  this  time,  the  purchase  price  for  the  LiveBoard  3.0  has  been  reduced  to  $34,900.  The  higher 
cost  here  is  a  five  year  lease-to-own  option  that  includes  a  maintenance  agreement. 

^he  unit  cost  here  is  a  five  year  lease-to-own  option  that  includes  the  basic  configuration,  optional 
features,  installation,  spares,  and  a  maintenance  agreement.  The  purchase  price  can  vary 
significantly  depending  on  the  number  of  student  stations  and  the  optional  features  included. 
Normally  a  one  year  warranty  applies  within  installation  done  locally. 

^These  costs  reflect  total  refurbishment  of  the  classroom  facility  space. 

®The  hardware  installation  costs  are  included  in  the  cost  elements  above. 

^Initial  instructor  training  was  provided  by  Lockheed  Martin  as  part  of  EETM  database 
development  effort. 

^The  course  conversion  costs  are  considered  implicit  costs  since  the  course  managers  and  costs  for 
the  conversion  effort  were  not  kept. 

4.3.1  Student  Station 

The  initial  configuration  for  the  student  station  assumed  that  each  station  would  operate  in 
a  stand-alone  mode.  However,  the  size  of  the  lETM  database  was  much  larger  than  originally 
anticipated  (10  CD-ROMs  versus  1  CD-ROM).  The  additional  CD-ROMs  posed  the  requirement 
for  some  type  of  expensive  jukebox  solution  to  present  the  database.  In  addition,  it  was  too  time 
consuming  to  load  the  software  on  each  system  in  the  classroom.  Therefore,  the  decision  was  made 
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to  set  up  a  local  area  network  (LAN)  to  link  the  instructor  station  and  student  stations  in  each 
classroom  so  only  the  instructor  station  (server)  has  to  updated.  The  initial  hardware  and  software 
configuration  for  the  student  station  has  been  upgraded  several  times  to  reflect  changes  in 
requirements.  For  example,  the  original  DX/2  66  MHz  CPUs  in  the  classrooms  were  upgraded  with 
no  instructional  downtime. 

The  current  hardware  configuration  for  a  student  station  includes: 

•  486  DX/4  100  MHz  CPU. 

•  16  megabyte  RAM. 

•  3.5  1.44  megabyte  floppy  drive. 

•  2.572  gigabyte  hard  drive. 

•  3  X  CD-ROM  drive. 

•  32  bit  Sound  Blaster  card. 

•  32  bit  LAN  card. 

•  17  SVGA  monitor. 

•  Keyboard. 

•  Mouse/Trackball. 

The  current  configuration  for  a  student  station  includes  the  following  COTS  software:  MS 
DOS  6.22,  Windows  for  Work  Groups  3.11,  MS  ACCESS  Run  time,  and  Guide  Reader.  No 
customized  software  is  required  to  view  the  lETM  Greeter  and  View  packages. 

The  student  work  station  consists  of  an  ergonomic  chair  at  a  flat  desk  with  a  glass  window 
for  viewing  the  monitor  mounted  under  the  desk.  The  CPU  and  the  keyboard  are  also  mounted 
under  the  desk  to  leave  a  clear  work  space  with  only  the  mouse  on  the  surface.  The  work  stations 
are  available  under  a  GSA  schedule. 

4.3.2  Instructor  Station 

The  initial  hardware  and  software  configuration  for  the  instructor  station  has  also  been 
upgraded  like  the  student  station  to  reflect  changes  in  requirements.  The  instructor  work  station 
consists  of  a  computer  desk  large  enough  to  accommodate  two  computer  systems  and  peripherals 
plus  at  least  one  ergonomic  chair  or  stool.  The  furniture  is  available  under  a  GSA  schedule. 

The  current  hardware  configuration  for  the  instructor  station  includes: 

•  486  DX/4  100  MHz  CPU. 

•  16  megabyte  RAM. 
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•  3.5  1.44  megabyte  floppy  drive. 

•  6  gigabyte  hard  drive. 

•  3  X  CD-ROM  drive. 

•  32  bit  Sound  Blaster  card. 

•  32  bit  LAN  card. 

•  Tape  Drive. 

•  20  SVGA  monitor. 

•  Keyboard. 

•  Mouse/Trackball. 

•  Laser  Printer. 

•  VCR. 

The  current  configuration  for  the  instructor  station  includes  the  following  COTS  software: 
MS  DOS  6.22,  Windows  for  Work  Groups  3.11,  MS  ACCESS,  Guide,  Video  for  Windows,  and 
SYTOS  tape  back-up  package.  No  customized  software  is  required  to  view  the  lETM  Greeter  and 
\fiew  packages.  Lockheed  Martin  also  provided  the  school  with  the  Audience  Tailored 
Instructional  Testing  Environment  (ATITE)  and  the  Test  Question  Analysis  System  (TQAS) 
software  as  part  of  the  lETM  database  development  effort.  No  documentation  or  user  training  was 
included  for  these  Navy  owned  software  packages. 

4.3.3  Local  Area  Network 

Each  of  the  GSE/GSM  automated  classrooms  includes  an  EtherNet  LAN  (300  feet  of  IEEE 
cabling  and  connectors)  to  connect  the  instructor  station  (server)  to  the  student  stations  (clients). 

4.3.4  LiveBoard 

The  Xerox  LiveBoard  is  an  electronic  white  board  designed  for  presentations  and  video 
teleconferencing.  It  features  a  67  inch  rear  projection  monitor  with  VGA  resolution  (640X480),  an 
Intel  486  DX/2  66  MHz  CPU,  a  wireless  remote  operation  pen,  full  motion  video,  and 
communications  software.  The  Windows-based  MeetingBoard  software  facilitates  connections  to 
peripherals  such  as  a  VCR  or  a  CD-ROM  player. 

The  LiveBoard  for  each  classroom  was  acquired  from  FSI  Products,  Inc.  on  a  5  year  lease- 
to-own  basis  with  an  extended  warranty  to  cover  the  lease  period.  Installation  was  included  as  part 
of  the  acquisition  costs.  Provisioning  for  spares  was  not  included. 
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4.3.5  Link  System 


The  LINK  Video  Networking  System  is  a  product  of  Applied  Computer  Systems,  Inc. 
(ACS).  The  Computer  Controlled  LINK  System  (CCLS)  installed  in  each  classroom  includes  the 
individual  student  control  units,  the  controller  (a  microprocessor,  a  system  logic  circuitry  board 
and  a  3.5  inch  disk  drive),  and  the  console  (a  14  color  monitor)  with  mouse  input.  The  following 
optional  features  are  also  included  in  the  classroom  configuration:  Projector  Control  to  connect  to 
the  Xerox  LiveBoard,  Magic  Pointer  and  Magic  Pointer  Control  to  display  free  style  drawing  on 
the  Xerox  LiveBoard,  and  LINK  Express  (a  question-answer-evaluation  system  that  provides  an 
instructor  with  immediate  student  feedback). 

The  system  is  hardware  driven  and  independent  of  software  or  data  LAN.  The  Link  System 
is  an  analog  overlay  to  the  video  signal.  It  features  an  on-screen  classroom  layout  with  student 
name  or  number,  flexibility  to  re-program  system  software  for  class  set-up  and  scheduling, 
automatic  initialization  and  self  diagnostics,  unlimited  connectivity,  as  well  as  plug-and-play 
installation  of  customized  cabling  routed  through  one  single  bus  cable. 

The  CCLS  is  the  second  system  at  the  instructor  station.  It  provides  the  instructor  with  the 
following  classroom  management  features: 

•  Receive— view  any  students  display  or  other  external  device  directly  on  the  instructors 
monitor. 

•  Transmit-send  the  instructors  display  to  any  or  all  student  monitors. 

•  Receive  and  Transmit-receive  one  students  display  or  external  device  on  the  instruc¬ 
tors  monitor  and  transmit  this  back  to  any  or  aU  student  monitors. 

•  Blank— send  a  blank  display  screen  to  any  one  or  all  student  monitors. 

•  Scan— monitor  every  student  display  sequentially  and  automatically. 

•  Swap— view  the  displays  from  any  two  student  monitors  by  toggling  from  one  image 
to  the  other  automatically  or  manually. 

•  Keyboard/Mouse  Control-take  over  the  operation  of  a  students  computer  by  using  the 
instructors  keyboard  and  mouse  when  receiving  that  students  display. 

•  Keyboard/Mouse  Freeze— freeze  any  or  all  student  keyboards  and  mice  to  stop 
unwanted  student  input. 

The  LINK  System  for  each  classroom  was  acquired  from  ACS  on  a  5  year  lease-to-own 
basis  with  an  extended  warranty  to  cover  the  lease  period.  Installation  was  included  as  part  of  the 
acquisition  cost.  The  system  is  highly  reliable,  but  provisioning  for  back-up  power  supply  and 
interface  units  is  suggested.  ACS  provides  replacements  for  faulty  components  by  overnight 
delivery. 
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4.3.6  Classroom  Preparation 


The  classroom  preparation  costs  for  the  five  GSEyOSM  automated  classrooms  are 
organized  into  the  three  categories  below.  The  costs  reflect  the  requirement  for  total  refurbishment 
of  the  facility  space.  Requirements  for  other  facility  spaces  may  not  be  as  extensive  or  costly  as 
those  shown  in  Table  4-5. 

•  Designs— includes  blueprints,  plans,  site  visits,  contract  bids,  etc. 

•  Computer  Room  Flooring— includes  acquisition  and  installation  of  metal  framing 
assemblies  to  keep  all  cabling  and  cords  under  the  floor. 

•  Installation— includes  demolition,  space  preservation  and  preparation,  electrical  power 
service  system  upgrades,  power  backup  unit,  electrical  power  conditioning  (ground- 
fault-isolation),  lighting  fixture  upgrades,  environmental  and  climate  control  systems 
(system  controls,  cooling/heating  equipment,  vent  ducting,  weather  seals,  registers/ 
diffusers,  dust/dirt  air  scrubbers,  anti-static  conditioners),  cable  wiring  and  connectors. 

4.3.7  Hardware  Installation 

Installation  costs  for  the  GSE/GSM  implementation  were  included  in  the  cost  elements 
described  above.  User  Training 

4.3.8  User  Training 

Lockheed/Martin  provided  initial  instructor  training  as  part  of  the  lETM  database 
development  and  implementation  effort. 

4.3.9  Course  Conversion 

Although  NAVSEA  was  responsible  for  developing  the  electronically  enhanced  IGs  and 
TGs  for  inclusion  in  the  lETM  database,  the  course  supervisors  and  instructional  staff,  with  support 
from  the  New  Technologies  Division  staff  at  SSC  GLARES,  were  responsible  for  rewriting  the 
paper-based  courses  for  conversion.  The  effort  to  redesign  the  courses  for  the  lETM/AC  format 
included  preparation  of  view  package  scripts  for  the  lETM  database,  development  of  revised  IGs 
and  TGs,  and  specification  and  development  of  on-line  instructional  materials.  Since  the 
conversion  effort  was  performed  by  in-house  staff  as  part  of  their  daily  activities,  no  data  was  kept 
on  the  level  of  effort  or  number  of  labor  hours  expended  to  initially  redesign  the  courses. 

The  revised  IGs  and  TGs  were  authored  using  AIM.  The  AIM  data  bases  were  then 
provided  to  Lockheed/Martin  to  extract  and  convert  the  training  materials  to  SGML  format  for 
inclusion  in  the  lETM  database.  The  costs  to  convert  the  IGs  and  TGs  to  SGML  format,  integrate 
them  into  the  lETM  database,  and  provide  the  materials  to  the  schoolhouse  on  CD-ROM  was 
included  as  part  of  Lockheed/Martins  overall  lETM  database  development  effort. 

Although  the  converted  IGs  and  TGs  were  integrated  into  the  lETM  database,  it  is 
important  to  note  that  the  instructional  materials  for  both  courses  continue  to  be  refined  and 
enhanced  for  use  in  the  automated  classrooms.  Initially,  the  lead  instructors  for  specific 
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instructional  units  assumed  the  responsibility  for  creating  a  standard  set  of  annotations  for  the 
revised  IGs.  The  individual  instructors  use  the  standard  set  of  annotations  and  then  add  annotations 
(often  in  the  form  of  visual  aids)  to  personalize  the  IG  as  they  see  necessary.  As  experience  is 
gained,  many  of  the  personal  annotations  and  visual  aids  are  being  incorporated  into  the 
standardized  annotation  set.  The  instructors  are  also  expanding  the  TGs  to  include  Information 
Sheets  in  addition  to  Assignment  Sheets.  Again,  no  cost  data  was  available  on  the  level  of  effort  or 
number  of  labor  hours  expended,  since  these  activities  are  an  integral  part  of  the  instructors  daily 
duties. 

The  quality  of  the  legacy  data  in  the  gas  turbine  technical  manuals  was  a  problem  for  both 
courses.  Converting  complex,  illegible  paper-based  graphics  and  multi-page  fold-out  schematics 
into  digital  format  often  did  not  provide  adequate  quality  for  classroom  presentation.  The  GSE 
course  is  currently  using  paper  hand-outs  for  multi-page  fold-out  schematics.  The  GSE 
instructional  staff  is  investigating  several  COTS  graphics  software  packages  for  developing  circuit 
diagrams  (i.e.,  SPICE  for  Windows  and  Circuit-Maker).  The  GSM  instructional  staff  has  developed 
a  number  of  visuals  for  classroom  presentation  using  draw  packages  and  various  COTS  graphics 
software.  They  have  also  developed  detailed  specifications  and  story  boards  for  complex 
animations  to  enhance  the  instruction.  The  animations  are  developed  by  a  contractor  under  an 
interactive  courseware  (ICW)  development  support  contract  managed  by  the  New  Technologies 
Division.  Since  this  contract  provides  ICW  development  support  across  SSC  training  activities, 
data  is  not  kept  on  the  number  of  labor  hours  expended  for  specific  projects.  Therefore,  an  estimate 
of  the  number  of  labor  hours  to  develop  the  GSM  animations  was  not  available. 

The  GSE  course  is  still  using  paper  tests  rather  than  on-line  testing.  The  GSM  instructors 
converted  the  paper  tests  to  an  on-line  multiple  choice  format  using  the  ATITE  and  TQAS  software 
provided  by  Lockheed/Martin.  Even  though  the  GSM  testing  has  been  automated,  the  interface 
with  the  Instructional  Support  System  (ISS)  is  a  manual  process. 

4.4  Life  Cycle  Cost  Model:  Recurring  Cost  Elements 

Based  on  the  data  collected  during  the  costs/benefits  analysis,  we  could  only  make  general 
statements  about  the  annual  support  requirements  for  the  GSE/GSM  classrooms.  These  general 
statements  are  summarized  below  in  terms  of  the  recurring  cost  elements  in  the  LCCM.  An  LCCM 
table,  like  the  one  shown  previously  in  Table  4,  was  not  prepared  since  we  could  not  develop 
recurring  cost  estimates.  Section  5.1.1  discusses  the  efforts  that  are  currently  in  progress  by  several 
Navy  organizations  to  define  the  life  cycle  management  requirements  for  the  lETM/AC 
technology. 

4.4.1  Hardware  Maintenance 

For  the  most  part,  the  computer  hardware  was  acquired  under  a  five  year  lease-to-own 
option  with  either  an  extended  warranty  or  maintenance  agreement  included  in  the  acquisition 
costs.  Since  the  configuration  is  based  on  COTS  hardware  it  tends  to  be  reliable  with  minimal 
downtime.  However,  additional  spares  provisioning  may  be  necessary  over  the  life  cycle  as  daily 
classroom  utilization  increases  from  one  or  two  shifts  to  three  shifts. 
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4.4.2  Software  Maintenance  and  Enhancements 

The  current  automated  classroom  configuration  does  not  include  any  customized  software 
other  than  the  Navy-owned  Al'lTE  and  TQAS  packages.  These  are  database  oriented  performance 
assessment  tools  that  the  user  can  tailor  to  meet  specific  testing  requirements.  If  these  tools 
continue  to  be  used,  no  software  maintenance  or  enhancements  are  anticipated. 

4.4.3  Hardware/Software  Upgrades 

The  costs  for  hardware  and  software  upgrades  that  have  already  been  installed  in  the 
instructor  stations  and  student  stations  are  included  in  the  acquisition  costs.  However,  additional 
hardware  and  software  upgrades  are  planned.  A  rule  of  thumb  that  may  be  applied  to  estimate  the 
costs  for  hardware/software  upgrades  is  ten  percent  (10%)  of  the  acquisition  costs  per  student 
station  and  instructor  station  will  be  expended  on  an  annual  basis. 

There  are  plans  to  upgrade  the  networking  software  to  Windows  NT  in  the  immediate 
future.  Since  the  Windows  for  Work  Groups  3.11  software  currently  in  use  is  a  peer-based  network, 
there  are  problems  in  maintaining  the  security  of  the  instructor  station  software  configuration  and 
on-line  testing.  Upgrades  to  other  COTS  software  are  also  expected  as  vendors  enhance 
capabilities  and  features. 

There  are  plans  to  upgrade  to  a  Pentium  processor  as  costs  come  down  and  more  courses 
come  on-line.  The  possibility  of  acquiring  removable  hard  drives  to  store  specific  course 
configurations  as  more  courses  share  the  automated  classroom  systems  has  also  been  discussed.  As 
technology  evolves  over  the  next  few  years,  any  number  of  hardware  upgrades  may  be  available 
to  enhance  capabilities. 

4.4.4  Logistical  Support 

Maintenance  of  the  hotel  services  (i.e.,  lighting,  electrical  power,  heating/air  conditioning) 
in  the  classrooms  is  the  responsibility  of  SSC  GLARES.  Both  preventive  maintenance  and  repairs 
should  be  handled  through  normal  organizational  services.  Preventive  maintenance  activities 
include,  but  are  not  limited  to,  periodic  testing  and  replacement  of  batteries  in  the  under  deck 
smoke  detectors,  inspecting  fire  extinguishers,  replacing  air  conditioning  filters  and  examining 
belts  for  wear,  replacing  light  fixtures 

In  addition  to  classroom  maintenance,  logistical  support  also  includes  the  costs  for  the 
following  activities: 

•  Scheduling  classroom  facilities. 

•  Maintaining  the  inventory  of  the  media/materials  required  to  support  a  course  in  a 
secure  space.  These  items  include  consumables,  supplies,  media  delivery  devices  and 
spare  components,  multiple  sets  of  technical  manuals,  training  aids,  instructional 
materials  and  tests. 

•  Reproducing  the  course  training  materials,  including  IGs,  TGs,  tests,  training  aids, 
other  training/reference  materials,  and  sets  of  technical  manuals. 
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•  Maintaining  security  and  accountability  in  the  distribution  of  instructional  materials 
and  tests  for  each  course  convene. 

For  the  most  part,  the  implementation  of  the  lETM/AC  technology  should  reduce  the  time 
and  effort  on  the  part  of  the  instructional  staff  and/or  support  personnel  to  perform  these  activities. 
However,  any  reduction  in  these  costs  are  more  than  offset  by  the  increase  in  costs  for  system/ 
network  administration  and  personnel  training  described  in  the  following  paragraphs. 

4.4.5  System/Network  Administration 

The  requirement  for  qualified  personnel  to  perform  system  administration  and  network 
administration  tasks  cannot  be  overstated.  Likewise,  the  effort  required  to  keep  the  lETM/AC 
technology  ready  for  classroom  operations  cannot  be  under  estimated.  System  administration 
includes,  but  is  not  limited  to,  the  following  activities: 

•  Provide  assistance  to  instructors  and  staff  in  solving  operational  glitches. 

•  Perform  preventive  maintenance,  diagnostics,  and  troubleshooting. 

•  Maintain  local  area  network. 

•  Maintain  hardware  and  software  configuration  management  control. 

Establish  policies  and  procedures. 

•  Schedule  classroom  utilization. 

•  Install  and  load  software. 

•  Manage  the  software  inventory. 

•  Maintain  physical  security  of  classrooms. 

•  Maintain  system  security. 

•  Maintain  system  backups. 

•  Interface  with  vendors  and  support  activities  as  required. 

A  rule  of  thumb  that  may  be  applied  to  estimate  the  level  of  effort  required  is  one  hour  per 
week  per  node  in  a  networked  system.  With  a  total  of  five  instructor  stations  and  65  student  stations 
in  the  five  GSE/GSM  classrooms  this  equates  to  approximately  two  full  time  personnel  to  perform 
system/network  administration  functions. 

4.4.6  User  Training 

Continual  training  programs  are  a  must,  considering  both  military  personnel  turnovers  and 
advances  in  technology.  Management  and  support  personnel,  as  well  as  instructors  need  to  be 
trained  to  effectively  use  and  manage  the  lETM/AC  technology.  Unless  a  civilian  certified  network 
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administrator  is  hired  or  provided  by  a  contractor,  formal  training  in  network  administration  is 
essential  for  personnel  assigned  to  that  task. 

4.4.7  Course  Revision 

As  described  previously,  the  GSE  and  GSM  courses  are  in  the  process  of  being  refined  as 
new  authoring  capabilities  and  techniques  are  investigated  as  part  of  the  R&D  effort.  After  the 
lETM-based  curriculum  is  approved  by  CNET,  then  course  revision  procedures  documented  in  the 
NAVEDTRA  131  training  manual  series  will  apply. 
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Discussion 


5.1  Implications  for  Future  Implementations 

An  important  aspect  of  the  effort  is  to  evaluate  the  impact  of  introducing  the  lETM/AC 
technology  into  the  schoolhouse,  identify  the  management  and  support  requirements  imposed  by 
the  technology,  and  develop  plans  to  transition  the  technology  from  the  R&D  environment  to  the 
training  community.  After  almost  two  years  of  experience,  a  number  of  problems  and  potential 
solutions  have  emerged  that  have  implications  for  the  costs  to  be  incurred  and  the  benefits  to  be 
realized  in  future  implementations.  The  following  paragraph  summarize  some  of  the  problems  and 
solutions  encountered  during  the  GSE/GSM  implementation. 

5.1.1  Life  Cycle  Support 

NAVSEA  tasked  the  following  organizations  to  define  the  life  cycle  management  and 
support  requirements  for  the  five  automated  classrooms  at  the  GS  C  School: 

•  Naval  Sea  Systems  Command  Automated  Data  Systems  Activity  (SEAADSA). 

•  Naval  Sea  Logistics  Center,  Code  30  (SEALOG). 

•  Fleet  Technical  Support  Center,  Atlantic  (FTSC-LANT). 

SEAADSA  was  designated  as  the  Hardware  In-Service  Engineering  Activity  (ISEA)  and 
the  Basic  Software  ISEA  for  the  classrooms.  The  term  hardware  is  broken  down  into  two 
definitions.  Computer  hardware  is  defined  as  any  piece  of  physical  equipment  including 
computers,  monitors,  desks,  and  chairs.  Non-computer  hardware  is  defined  as  any  piece  of  physical 
equipment  related  to  hotel  services  (i.e.,  lighting,  electrical  power,  heating/air  conditioning)  that  is 
necessary  for  the  proper  operation  of  the  lETM/AC  configuration.  Basic  software  is  defined  as  any 
computer  program  or  collective  group  of  programs  that  contribute  to  the  operation  of  the  lETM/ 
AC  (i.e.,  Windows  and  other  standard  software  that  may  be  loaded  on  the  system  at  the  time  of 
purchase).  SEAADSA  is  responsible  for  coordinating  any  changes  in  the  hardware  and  basic 
software  configuration  for  the  SSC  GLAKES  automated  classrooms. 

SEALOG  was  designated  as  the  Hardware  and  Basic  Software  Life  Cycle  Manager 
(Hardware  LCM).  Personnel  are  in  the  process  of  preparing  a  Life  Cycle  Management  Plan  that 
documents  the  responsibilities  and  tasks  for  supporting  the  classrooms  over  their  life  cycle.  Since 
most  of  the  computer  hardware  was  acquired  on  a  five  year  lease-to-own  basis  with  five  year 
service  agreements,  a  life  cycle  of  five  years  is  assumed  for  the  current  configuration.  The  decision 
to  retain,  upgrade,  or  replace  the  computer  hardware  will  be  made  jointly  by  SSC  GLAKES, 
SEAADSA,  and  SEALOG. 

FTSC-LANT  was  designated  as  the  Software  ISEA.  Personnel  are  in  the  process  of 
preparing  a  Software  Life  Cycle  Management  Plan  that  documents  the  responsibilities  and  tasks 
for  supporting  the  software  configuration.  The  software  considerations  include  the  Annotation 
Manager,  the  lETM  database,  and  ICW. 
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SSC  GLARES,  as  the  end  user  of  the  lETM/AC  technology,  is  responsible  for  monitoring 
the  condition  of  the  hardware  and  software  configuration,  ensuring  appropriate  maintenance 
actions  are  taken,  and  coordinating  with  SEAADSA,  SEALOG,  and  FTSC-LANT  on  life  cycle 
support  issues  for  the  classrooms.  Based  on  the  experience  of  the  R&D  effort,  it  is  clear  that 
implementation  of  the  technology  imposes  additional  personnel  support  requirements  on  the 
schoolhouse.  Some  personnel  support  has  been  provided  by  vendors  and  contractors  involved  in 
the  effort.  However,  most  of  the  personnel  support  for  the  automated  classrooms  has  been  from  the 
school  managers,  course  supervisors  and  instructors  with  assistance  from  the  New  Technologies 
Division  staff.  Therefore,  the  GS  “C”  School  managers  are  preparing  an  ASDP  for  submission  to 
NETPMSA  to  address  the  additional  support  requirements  over  the  life  cycle  of  the  lETM 
automated  classrooms. 

5.1.2  Acquisition  Planning 

Adequate  planning  is  key  to  successful  implementation  of  the  lETM/AC  technology.  As 
documented  in  a  paper  presented  by  the  Gas  Turbine  Technical  Training  Officer,  LT.  Eric  Fretz 
(1995),  the  speed  with  which  the  technology  was  implemented  in  the  GSE/GSM  courses  led  to 
some  early  mistakes  that  could  be  avoided  by  more  planning  and  discussion.  For  example,  the  size 
of  an  automated  classroom  was  decreased  to  a  maximum  of  12  or  15  students  from  the  maximum 
of  25  students  in  a  paper-based  classroom.  The  classroom  size  has  implications  for  supporting  the 
projected  student  throughput  as  well  as  classroom  scheduling  and  system  resource  management. 
The  scheduling  for  the  lETM  classrooms  follows  the  Navy  6+2  Initiative.  Six  hour  classroom 
sessions  are  from  0600  to  1200  and  from  1800  to  2400.  The  two  hours  of  either  mandatory  or 
voluntary  study  time  are  available  from  1300  to  1700  for  students  from  either  session.  To  meet 
projected  throughput  requirements,  the  GSE  school  has  three  classrooms  with  12  student  stations 
each,  the  GSM  school  has  one  classroom  with  15  student  stations,  and  both  schools  share  one 
classroom  with  12  student  stations. 

Also,  the  initial  hardware  and  software  configuration  did  not  include  the  use  of  a  local  area 
network  (LAN)  to  link  the  instructor  station  (server)  and  the  student  stations  (clients)  in  the 
classroom.  When  this  requirement  was  recognized,  an  EtherNet  LAN  and  Windows  for  Work 
Groups  3.11  were  installed.  However,  this  simple  setup  does  not  provide  the  necessary  capabilities 
and  security  features  so  plans  are  currently  being  made  to  upgrade  to  Windows  NT  software.  In 
addition  to  the  issue  of  maintaining  the  security  of  the  course  tests  and  the  instructor  station 
configuration,  physical  security  of  the  classrooms  and  management  of  the  COTS  software 
inventory  are  major  concerns  that  need  to  be  addressed. 

Although  adequate  planning  is  essential,  the  process  normally  required  for  planning  and 
implementing  such  technology  is  too  time  consuming.  Often  times  the  technology  is  obsolete 
before  the  system  is  installed.  Management  tools  need  to  be  developed  to  support  planning  and 
budgeting  phase  and  facilitate  the  acquisition  and  installation  phase.  NAVSEA  has  tasked 
personnel  at  FTSC-LANT  to  develop  a  generic  Equipment  Facilities  Requirement  Plan  (EFRP) 
and  a  generic  Abbreviated  System  Decision  Paper  (ASDP)  for  use  in  planning  and  initiating  future 
implementations.  The  purpose  of  these  generic  documents  is  to  identify  the  basic  requirements  and 
indicate  where  school  specific  information  needs  to  be  furnished. 
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Personnel  from  SEAADSA  discussed  the  requirement  to  streamline  the  process  for 
procuring  the  automated  classroom  components.  Efforts  are  underway  to  establish  a  delivery  order 
type  contract  to  allow  schools  to  acquire  COTS  hardware  and  software  (i.e.,  the  LINK  System  and 
the  LiveBoard)  that  are  specified  as  part  of  the  configuration  and  are  not  available  under  the  current 
small  multi-user  computer  contract  or  GSA  schedules.  This  type  of  contract  may  include 
provisions  for  quantity  buys,  spares,  installation  and  maintenance  support,  and  user  training.  It 
should  result  in  lower  acquisition  costs  than  those  incurred  for  the  R&D  effort  and  it  should  reduce 
the  lead  time  to  order  and  install  the  equipment. 

5.1.3  User  Training 

As  described  earlier,  user  training  is  critical  for  successful  implementation  of  the  lETM/AC 
technology.  Appropriate  user  training  must  be  provided  for  students  and  instructors,  as  well  as 
course  management  and  support  personnel.  The  need  for  qualified  personnel  to  perform  system 
administration  tasks  and,  especially,  network  administration  tasks  cannot  be  over  emphasized.  As 
stated  by  Fretz  (1995): 

For  now,  the  resident  computer  geek  simply  learns  on  the  job  and  from  ven¬ 
dors/installers.  This  corporate  knowledge  is  perishable  and  puts  the  school 
up  against  a  rock  when  the  military  member  transfers.  It  is  possible  to  ame¬ 
liorate  this  situation  with  a  GS/contracted  network  administrator,  but  contin¬ 
ual  training  programs  are  a  must.  (pg.  2) 

If  qualified  civilian  or  contractor  personnel  are  not  hired,  then  formal  training  programs  in 
microcomputer  hardware  and  software  configuration  management  and  certification  as  a  network 
administrator  are  essential  for  military  personnel  assigned  those  duties. 

Instructors  are  the  primary  source  of  innovation  and  key  to  the  success  of  the  system.  They 
need  to  be  involved  in  the  process  of  creating  the  end  product  for  use  in  the  classroom.  As  part  of 
the  lessons  learned,  weekly  instructor  training  programs  are  scheduled  to  share  corporate 
knowledge,  float  ideas,  and  demonstrate  proven  concepts  (Fretz,  1995). 

To  maximize  the  effectiveness  of  the  lETM/AC  technology,  the  instructors  must  be 
adequately  trained  on  the  operational  features  and  enhanced  capabilities  of  the  technology.  Since 
the  instructional  staff  has  varying  degrees  of  computer  literacy  and  experience,  the  initial  training 
was  provided  on  an  ad  hoc  basis  by  the  instructors  with  computer  experience  who  were  willing  to 
experiment  with  the  system.  Based  on  this  experience,  the  GSM  course  managers  at  SSC  GLARES 
have  developed  a  40  hour  syllabus  for  initial  instructor  training  on  how  to  use  the  lETM/AC 
technology  in  the  classroom.  This  syllabus  will  be  used  to  train  incoming  GSM  instructors  and  it 
is  being  modified  to  train  GSE  course  instructors. 

Members  of  the  instructional  staff  may  also  require  training  in  COTS  applications  software 
that  is  used  to  author  the  instructional  materials  and  development  multimedia  training  aids.  As 
requirements  and  enhancements  are  defined  and  demonstration  software  packages  are  evaluated, 
user  training  should  also  be  considered  to  reduce  the  learning  curve  and  increase  productivity. 
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On  the  first  day  of  class,  students  receive  approximately  6  hours  of  training  on  how  to  use 
the  lETM/AC  technology.  The  training  varies  depending  on  the  students  familiarity  with 
computers.  Novices  are  directed  to  go  through  the  mouse  and  Windows  tutorials  before  being 
introduced  to  the  lETM  on-line  help  index  and  practicing  navigating  through  the  lETM.  Students 
concentrate  on  searching  the  database,  making  annotations,  and  creating  bookmarks.  Instructors 
report  that  most  students  are  comfortable  with  the  system  and  proficient  in  using  it  by  the  end  of 
the  first  week  of  class.  Students  that  continue  to  have  trouble  may  receive  additional  assistance 
during  the  voluntary  study  time. 

Students  have  the  capability  to  use  the  annotation  feature  in  the  lETM  database  to  take 
notes  on-line  at  their  work  stations  during  class  sessions.  However,  there  appear  to  be  several 
drawbacks  to  on-line  note  taking.  Many  students  do  not  have  basic  typing  skills  and  hunting  and 
pecking  on  the  keyboard  disrupts  their  concentration  on  the  course  content.  Since  the  only  systems 
available  for  students  to  review  their  notes  on  floppy  disk  are  the  ones  in  the  lETM  classrooms, 
their  study  time  may  be  restricted  to  the  afternoon  hours.  Systems  may  not  always  be  accessible 
after  hours  or  over  weekends.  Likewise,  the  students  may  not  be  able  to  review  their  notes  on-line 
after  they  return  to  their  jobs  in  the  fleet. 

5.1.4  Interface  with  lETM  Database 

The  Naval  Warfare  Surface  Center,  Carderock  Division,  Philadelphia  Detachment 
(NAVSSES)  is  the  Gas  Turbine  In-Service  Engineering  Activity  (GS ISEA).  As  such,  this  group  is 
responsible  for  managing  the  GS  technical  manuals  and  providing  technical  expertise.  They 
worked  closely  with  Lockheed/Martin  in  the  conversion  of  the  technical  manuals  to  the  GS  lETM 
database.  They  also  participated  in  the  development  of  the  view  packages  for  the  GS  lETM 
database  with  assistance  from  the  instructional  staff  and  New  Technologies  Division  personnel  at 
SSC  GLARES.  Even  though  Lockheed/Martin  was  responsible  for  developing  and  installing  the 
GS  lETM  database  in  the  fleet  and  in  the  schoolhouse,  NAVSSES  will  assume  responsibihty  for 
maintaining,  updating,  and  disseminating  the  database  in  the  future. 

NAVSSES  is  currently  in  the  process  of  establishing  a  hardware  and  software  configuration 
to  duplicate  the  lETM  development  facility  used  by  Lockheed/Martin.  They  are  also  investigating 
techniques  for  improving  the  quality  of  the  database.  Techniques  include  better  quality  control  of 
legacy  data,  improved  scanning  capability,  converting  raster  scan  images  to  a  digital  (ASCII/ 
vector)  format,  and  evaluating  the  value  added  by  view  packages  versus  other  expert  models  that 
may  be  developed  to  guide  the  user  through  the  technical  content. 

The  GSE  and  GSM  course  IGs  and  TGs  authored  using  AIM  were  integrated  into  the  lETM 
database  and  the  link  with  AIM  was  lost.  The  links  between  lETM  databases  and  other  major  Navy 
systems  like  AIM  need  to  be  analyzed  to  allow  data  from  diverse  sources  to  be  dynamically 
managed.  The  schoolhouses  need  to  retain  control  of  the  IGs  and  TGs  and  the  course  revision 
process  needs  to  independent  of  the  BETM  development  and  revision  process.  However,  there 
needs  to  be  a  two-way  flow  where  updates  in  one  set  of  data  flag  changes  in  the  corresponding  areas 
of  the  other  set  of  data  to  minimize  the  labor-intensive  manual  process. 

The  plan  is  to  disseminate  updates  to  users  via  CD-ROM  on  a  semi-annual  or  possibly 
quarterly  basis.  Updates  will  include  changes  in  the  technical  manuals  as  well  as  database  fixes 
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such  as  failed  links  and  text  and  graphics  errors  that  users  identify.  A  system  needs  to  be  developed 
like  the  TechMan  Deficiency  Reports  used  for  paper  technical  manuals  to  document  deficiencies 
in  the  database.  The  technology  should  facilitate  receiving  inputs  from  both  the  fleet  and  the 
schoolhouse  and  disseminating  corrections  to  the  end  users. 

5.2  Potential  Benefits 

The  statement  of  benefits  presented  as  part  of  the  costs/benefits  model  addressed  the 
immediate  benefits  that  could  be  analyzed  and  quantified  in  a  meaningful  way  for  the  GSE/GSM 
demonstration.  However,  a  variety  of  other  benefits  may  be  realized  in  future  implementations  of 
the  technology  in  other  training  environments.  Some  potential  benefits  are  summarized  below. 

Benefits  in  both  preparing  and  delivering  instruction  may  be  realized  by  incorporating  the 
Training  Integration  Management  Software  (TIMS),  currently  under  evaluation  in  a  Diesel 
Division  course  at  SSC  GLARES,  into  the  configuration  for  future  lETM  automated  classrooms. 
This  format  neutral  capability  allows  both  PPP-based  instructional  materials  authored  using  the 
current  version  of  AIM  and  task-based  instructional  materials  authored  using  Word  Perfect  to  be 
used  in  the  automated  classrooms  without  the  need  to  convert  the  materials  to  SGML.  Significant 
cost  savings  may  be  achieved  by  eliminating  the  need  to  translate  instructional  materials  into 
SGML  format. 

TIMS  includes  several  features  to  facilitate  instructor  management  of  the  course 
presentation.  The  software  has  a  broadcast  feature  to  allow  digital  sharing  of  displays  in  addition 
to  the  analog  video  linking  system  currently  implemented  in  the  lETM  classrooms.  It  also 
incorporates  a  programmable  remote  control  to  allow  the  instructor  to  control  the  presentation  from 
anywhere  in  the  classroom.  Although  these  features  do  not  necessarily  have  cost  implications,  they 
may  represent  gains  in  efficiency  in  managing  the  classroom  environment. 

Another  potential  benefit  may  be  the  integration  of  automated  performance  assessment 
tools  into  the  lETM/AC  software  configuration.  Both  the  curriculum  developers  and  the  classroom 
instructors  could  benefit  from  automated  test  generation  and  scoring  as  well  as  analysis  and 
recording  of  student  performance.  As  advances  in  technology  are  made,  it  may  be  possible  to 
streamline  the  process  and  integrate  the  results  of  automated  performance  assessment  software 
with  the  reporting  requirements  of  the  Instructional  Support  System  (ISS). 

As  documented  by  Fretz  (1995),  the  GSE/GSM  implementation  uses  the  classic  paradigm 
for  group  paced  instruction.  The  instruction  is  presented  via  instructor-led  lectures  with  the 
technology  providing  computer-controlled  multimedia  support.  Students  have  on-line  access  to  the 
TGs  and  lETM  database,  as  well  as  an  on-line  capability  to  take  notes,  but  the  instruction  is 
presented  in  a  lock  step  mode.  If  the  lETM/AC  technology  is  integrated  with  self-paced 
instructional  modules  (ICW),  it  may  be  possible  to  further  reduce  the  overall  training  time  and 
student  training  costs,  as  well  as  increase  operational  readiness. 
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Section  Six 


Recommendations 


6.0  Recommendations 


1.  The  GSE/GSM  demonstration  highlights  the  need  for  coordination  among  a  number  of 
Navy  commands  and  organizations.  As  the  technology  evolves  and  costs  decrease,  more  systems 
in  the  fleet  will  be  targeted  for  lETM  development  and  more  formal  courses  in  the  schoolhouses 
will  be  identified  for  conversion  to  an  lETM/AC  format.  However,  the  roles  and  responsibilities 
of  the  cognizant  organizations  (Systems  Commands,  ISEA  and  other  fleet  support  activities, 
CNET,  NETPMSA  and  other  training  support  activities,  and  the  NAVEDTRACOM  schools) 
should  be  better  defined  and  procedures  should  be  established  to  ensure  coordinated  efforts  that 
result  in  the  cost  effective  implementation  of  the  lETM/AC  technology  in  the  fleet  and  in  the 
schoolhouse. 

2.  To  accommodate  the  Program  Objective  Memorandum  (POM)  budgeting  cycle,  the 
current  time  line  for  submission  of  both  an  Equipment  Facilities  Requirement  Plan  (EFRP)  and  an 
Abbreviated  System  Decision  Paper  (ASDP)  is  six  years  prior  to  project  implementation.  Consid¬ 
ering  the  rapid  advances  in  technology,  the  trends  towards  obsolescence  in  a  few  years,  and  the 
evolving  nature  of  lETM  processes  and  products  (as  documented  in  the  survey  of  lETM  practices 
conducted  by  Kribs  &  Mark,  1995),  a  shorter  time  frame  between  project  initiation  and  subse¬ 
quent  funding  and  implementation  should  be  established. 

3.  With  the  continuing  introduction  of  technology  in  the  fleet  and  in  the  schoolhouse,  the 
methods  and  media  for  training  are  changing  and  the  roles  and  responsibilities  of  instructors  are 
evolving.  The  formal  Instructor  Training  Schools  should  be  revised  to  provide  training  in  topics 
such  as  computer  literacy  and  operator  skills,  techniques  for  designing  and  presenting  computer- 
based  instructional  media,  use  of  automated  performance  assessment  tools,  and  technology- 
driven  instructional  strategies. 

4.  The  automated  classroom  technology  imposes  new.  skill  requirements  for  management 
and  support  personnel.  Qualified  personnel  are  needed  to  perform  both  the  system  administration 
and  network  administration  tasks.  The  possibility  of  establishing  a  formal  school  to  produce  a 
Navy  version  of  an  industry  certified  information  system  manager  and  network  administrator 
should  be  investigated. 
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Chief  of  Naval  Education  and  Training  (Code  00),  (Code  04)  (2),  (Code  N5),  (Code  LOl) 
(12),  (Code  03EE1)  (2),  (Code  N7)  (2) 

Commanding  Officer,  Reet  Anti-Submarine  Warfare  Training  Center,  Atlantic 
Commanding  Officer,  Submarine  Training  Facility,  San  Diego 
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Commanding  Officer,  Reet  Training  Center  (Code  00),  San  Diego 
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Commanding  Officer,  Reet  Training  Center,  Naval  Station,  San  Diego  (Code  021.2) 
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COMTRALANT  (Code  01  A) 

COMTRAPAC  (Code  N-31) 

Pentagon  Library 
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